There's a room in his mind palace he fears even more than the padded cell where the ghost of Moriarty is locked in. It's a dark room without windows, with nothing in it but the image of his teenage brother huddled in a corner, crying his heart out.
That's the one time he saw his brother crying, and it's still haunting him thirty years later. He's never been able to determine what caused that moment of weakness, but not in a thousand years he's going to forget it.
It's the burning reminder that even Mycroft is only human after all.