The thing is; Sam could have had it all.
He could have had the white picket fence, the beautiful wife, the apple pie life. He could have had children, maybe even a dog who greeted him when he came home.
And he doesn't have it all.
And it just feels so wrong to Dean.
His brother could've been happy, if he hadn't gone to Stanford that night, asking Sam for help to find Dad, if Dad hadn't gone 'missing', if they had never gotten in the business of hunting.
Sam had almost had it all.
And now he didn't have anything besides Dean, and while it didn't seem like much, it was enough for Sam, even if he had wanted the apple pie life, had almost had the apple pie life before it was ripped so cruelly away from him.
Because Dean could always make everything better, no matter what happened.
Even after Jess was killed, and John sold his soul, and Sam was stabbed and killed, Dean is always there to make things better.
So maybe Sam doesn't have it all, but he has Dean, and Dean is enough.
