They'll never trust him.
In that way, he thinks he might understand now a little of how Agent Romanoff feels. Once an enemy, always an enemy, no matter how many times you've proven yourself worthy. Even his own 'brother' will probably never really trust him again. He'll always be looking out for signs of Loki's betrayal.
It shouldn't hurt him so much. After all, what he did was wrong. He sees that now. But his brother, his lover, the only person in all of the realms he can honestly say he cares for... Knowing that Thor will never trust him again tears him apart inside.
Thor denies it. Of course he does; he loves Loki. But Loki is the god of mischief and lies, and he can damn well tell a lie when he hears one.
He's tired. He's so tired of being watched closely at all times, tired of everyone suspecting him to be behind the latest enemy, no matter how many times he proves himself a friend.
And when he and Natasha get into an argument, just a small one over some small mistake he made during a battle, she gives him the sweetest smile he's ever seen on her cold face and says,
"It's not so easy, wiping your ledger clean, is it?"
And just like that, he understands how truly alike they are.