This is the hardest thing Castiel has ever done. The apocalypse, the fall, losing his grace, none of it compared. The way Castiel felt then was a universe apart to the stabbing pain he was feeling right now. Somewhere below his ribcage, there was a hole. A hole that had, until recently, been filled by Dean Winchester.
Castiel stood there on the bridge in the rain and watched Dean walk away from him. He didn't follow. He knew he couldn't. His loyalty to Dean was what made him stay there, with Sam. Castiel knew that Dean would never forgive him if he left Sam alone. Castiel would never forgive himself. He had become close to both brothers and would not allow Sam to come to harm because of his own selfishness. Dean had to go, and Castiel had to stay. He had to stay and watch the better half of him walk away.
"I guess what the humans say is true," Castiel murmured. "Heart's really do break."