Dean Doesn't Trust Sam
Dean didn't trust Sam.
That's not to say he didn't love the kid. You know the saying there's no love without trust? It's crap; Dean's living proof of that now.
It's not because Sam started the Apocalypse. It's not because of the demon blood drinking. Dean knew Sam meant well by drinking demon blood; he'd wanted to save the world, kind of. Either way Sam didn't set out to be evil. The kid still cries whenever Bambi's mom dies, he isn't evil. The Apocalypse thing? That was an accident, a mistake. Guess what? People make them. Dean would have taken down Lilith if he had a shot too; no one knew. This also wasn't about killing Lilith either.
This wasn't about Sam attacking Dean and Bobby and almost killing Dean. Dean was still hurt that Sam did it but they were demon hunting guys; they fought and from time to time it got out of control. You would not believe the trouble they got into with John for fighting when they were younger. Sure, Dean was seriously pissed off, but he'd even called Sam and apologized right afterwards. The choking thing wasn't what all this was about. Hell, it wasn't even about that bitch Ruby. It hurt like hell that Sam would choose her over him. It hurt worse that Sam seemed to think it was Dean's fault for smothering him, but that had nothing to do with Dean's trust issues. Okay, that's a lie it had a little to do with it, but that was only where demons were concerned.
This also had absolutely nothing to do with Sam being a freak. Yes, he said it; Sam is a freak. Come on, the two of them hunted ghosts for a living and Dean enjoyed the sight of a decapitated vamp way more than he should. They were both freaks. In fact, everyone's a freak; Sam's just a more delicate case. He couldn't help it though and Dean never ever blamed Sam for that.
No, the actual reason was that Dean was first and foremost a hunter. His hunter instincts told him not to trust Sam. If he was honest with himself it might have had a tiny bit to do with all the above things Sam's done, but only a teeny tiny bit. The main issue was Sam himself. Not his blood of his blood sucking or his mistakes, just Sam. Sam was angry and temperamental; he didn't know his place in life. The fact is, no one gives you a place in life; it's your place to go and get your own, but Sam? He wanted someone to hand him his place and the answers, and when he didn't get it he got angry. That's why he loved Ruby so much. She offered him a place as a hero. How could Dean trust someone that only knew how to throw tantrums and be insecure?
However, despite all of that there was an even bigger reason. Sam didn't trust himself or trust (or even knew) what he wanted. Sam thought he was a freak. He watched his own moved carefully. He never listened to his instincts and that was what a hunter, or anyone with their lives, counted on the most. Sam bled a woman dry and Dean knew enough about the kid to know that it killed him on the inside. Everyone that Sam ever trusted, everything he was taught, everything he felt, every nerve in his body was screaming no, and Sam did it anyways. He still drank the blood, still went with Ruby, and still bled an innocent woman dry. That bothered Dean way more than letting Lucifer out. That had been an accident; sucking the juice out of an innocent woman was not. To trust someone who didn't trust themselves or their instincts was like a death wish, even for regular people. Even more than that Sam didn't know what he wanted, and even when he had a few moments of clarity and did he didn't go and get it. Sam told Dean about his extreme doubts and all of the hallucinations telling him no; this only made Dean madder. Sam's hallucinations were what he really thought and by the number of them telling him that demon blood was a bad trip Dean could tell Sam wasn't sure he was right at the time. Sam bet his soul and what would have been his life on something he wasn't 100% sure about, hell; he wasn't even really 50% sure about. And at the warehouse? Sam wanted to back down but he didn't because he was too scared that wasn't what he wanted. Its fine if he didn't know, plenty of people didn't know what they wanted, but Sam was an interesting and dangerous type of person because he had no clue what he wanted, but was sure as hell going to try to take it anyways. He was like a chicken with its head cut off, but this time the chicken had an axe and was just running around chopping people up. Sam wanted to avoid the Apocalypse so he killed Lilith, but it was also becaue he was sad Dean died, or he offed her because of some revenge thing, or it was to protect Dean, but it was also because Dean smothered him and Sam needed to get away. Dean didn't doubt all of them were the truth, and that's what made Sam untrustable. He had no clear goal, but was going to go grab one anyways. Sam wanted to be an adult, but expected Dean to stay and fix things, or he wanted to go have alone time, but always asked to come back before Dean asked him to. Even now the kid still isn't sure whether or not he wants demon blood, but he refuses help and will fight Dean tooth and nail on the subject in an attempt to prove he knows what's going on.
That's why Dean didn't trust Sam. Sure the Apocalypse and the choosing Ruby thing didn't help, but if Dean was to tell the truth, he didn't trust Sam before this. It only really came to light now because Dean said it out loud, that and he's pissed now. Sam never trusted himself, was always insecure, and never knew what he wanted. How can Dean trust him?
So Dean loves Sam; sure he's mad, but that'll pass eventually. He loves him, but until Sam figures himself out there is no way Dean can trust him.