It wasn't a carefully guarded secret that the Brandybuck family was a little… well queer. What was a carefully guarded secret was the nature of what made the family queer in the first place. No one but the family in question and their spouses and children knew.
What made the family so queer was the fact that they could see spirits and ghosts. From the friendly ghost that wanted to say good bye to their remaining family and moved on after a day to the malicious spirits and ghouls that attack and harm the living, they could see it all.
Most of the Brandybucks don't have enough power to properly see the spirits; rather they could feel them in the vicinity.
Frodo Baggins was half-Brandybuck on his mother's side, and in him he had the power to see and hear the spirits. It was a level of the gift that hadn't been seen in years. Frodo from a young age understood what ghosts were and was taught how to exorcise them if they turned malicious and on him.
It was for this reason that when his parents died drowning and came to him as specters dripping in the water of the river they had died in and said their goodbyes he understood. Death was not something he did not know about. It was something he was taught from a young age.
So, he said his goodbyes and that was that. They were gone and there was nothing he could do about it.
When his uncles told him he was to go live with an Uncle on the Baggins side of the family he was hesitant at first. While he loved Uncle Bilbo, he didn't know of his ability. It was a closely guarded secret after all (one that could potentially when he hit maturity bump him up in line to head of the Brandybuck family. But that would be forsaking the Baggins name and taking his mother's name and he did not know if he could do that just yet.)
But he agreed and off he went to live at Bag End.
He just wasn't expecting the 3 Dwarf Ghosts to be haunting the smial.