He always wondered if his son would have been better off growing up with a sick grandmother than a depressed (and somewhat neglectful) father.
The boy was surrounded with an odd variety of people during his early years. His son was around him to most, of course, and he truly did his best to raise him. Then Marcellus had Haymitch, who would come to check in every week, and who was like a drunken uncle that never really paid any attention to him.
Also, the boy had Primrose Everdeen who (probably against everyone's wishes) stopped in every other day to make sure both he and his son were well. She taught him how to take care of Marcellus and later watched the child when he did have to go to the Capitol for the Games. Prim was like his son's nanny and he would have paid her for it if he had not already known she had access to everything he could possibly give her.
It was hard not to love Prim, and he knew both he and his son did love her. But he also knew he could (would) never be in love with Prim like he was (still) with her sister.
(She's like the sister I could (would) have had, He thought to himself some days).