It hits Sam when Dean's been in the frying pan about a week.
Dad had gone through Hell, too.
That's what it is, and it makes Sam sick to think that a) he's never thought about it, b) it makes him angry that it's another thing that Dad and Dean share that he doesn't get to be apart of and c) he's permanetly alone, now.
Well... he has Bobby. Sam doesn't plan on ever talking to Bobby ever again, though, so the old hunter doesn't count.
And Ruby...
Ruby's there with him for the sole reason of her blood (and, okay, the sex is great, but not the point.)
So, really, he's alone in the world.
Dean left him.
Dad left him.
Mom left him.
He's leaving Bobby, because that's what his family does. Leave.
Leaving must be in his blood.
Just like Azazel's, and Ruby's.
Well, it doesn't bother him as much as it should, then.