It was a year later (the winter after the 79th Hunger Games, the games he stayed locked away in the monitoring room for) that the bakery burned down, both of his parents dying with it. It was considered a tragic accident. After that he rarely went anywhere, just his house and Haymitch's.
His two brothers rebuilt the place (with his money) and worked there, but he refused to step foot in the new building. After time, he eventually lost touch with his brothers.
He always blamed (and would always blame) himself for his parents' death. He remembered Finnick's story about having to act for the Capitol all to well and convinced himself the fire was no coincidence.
He had refused to party, to talk, to smile, to even show himself the last time he went to the Capitol. They had missed him sorely. And although there was no evidence that the Capitol had anything to do with the fire, nothing to prove his theory right, he took it as a warning.
The next year he went back and partied with Finnick and was adored by the Capitol.
(The fire was the only horrible accident he ever saw after that).