It wasn't always like this. Baelfire constantly reminds himself of this. There's some stubborn – or hopeful – part of him that believes it could somehow go back to the way that it was before. Not before his mother left – he barely remembers her, but he does remember her making Papa miserable – just… before Papa took on the curse.
He was happy then; he had friends then, even if he didn't have fine things like he does now! He just wishes that he could get Papa to see it that way.
Or maybe he could just make a deal with him instead!