The Ponds were gone. They were gone. The Doctor said it a thousand and one times; each time, it sounded like another language. They couldn't just be gone. The Ponds were as much of a constant as the TARDIS. They were always there when it mattered, ready to fly away with him into the stars.
The angels had ripped a hole in the universe the size of Amy and Rory. The Ponds had had a life back home in 2012, with their flat and their family and their friends. No one would ever know what had happened to them. No one would know how brave they had been, right up until the end. No one would know of the sacrifices they made for the world and for each other.
The Doctor went back and told Brian, because he owed him that much. The Doctor told him he could stop watering the plants. He told him that Amy and Rory were happy, and that they wouldn't be coming back. They could never come back. Brian was angry at first – who wouldn't be? The Doctor had failed him, like he had failed Amy and Rory. But after the initial shock, Brian just sort of collapsed in on himself. After all, he was the one who had told them it was okay to go with the Doctor. He had known the danger, and he had let them go anyway. He blamed himself more than he blamed the Doctor. The Doctor placed blame on himself and himself alone. Both of them were crippled with the survivor's guilt.
Brian never quite got past the loss of Amy and Rory. It didn't help that he couldn't talk about it with anyone. He bottled it up for the longest time. In the end, the depression consumed him. The Doctor maybe could have helped, but he was also being smothered by the loss.
That's what it was: a big, smothering presence. The loss wasn't a hole at all – it was a big, heavy entity that hung over the Doctor's head wherever he went. He had companions on his adventures from time to time, but none of them ever went in the TARDIS. He couldn't bring himself to take that step. The people he met, though, could always tell that he had lost someone. Or rather, two someones. The loss of the Ponds was a tangible presence in the Doctor's life, and it wasn't one that was likely to go away anytime soon.
After all, they were seared onto his hearts.