He wishes he could have been strong.
Once she stepped up onto the makeshift stage in the square of District 12, he felt weak, helpless. All he wanted to do was lie down on the forest floor under the cover of trees and slip into oblivion. To never have to hear another voice, see another person, feel the touch of anyone's coal-dusted skin. To never have to smell the distinct smoke that marked an accident in the mines or taste the grime in the air that followed. To never have to think of the Hunger Games or the Capitol. To be nothing.
If he were stronger, he could've left with her months before the Reaping. She wouldn't have to fight anyone. It would just be the two of them and the forest, nothing else to mar their lives. They could've been free.
If he were stronger, he could've told her he loved her. Their goodbye wouldn't have been so empty and silent. She wouldn't have pretended to love Peeta Mellark. Even if she didn't love him back, she wouldn't throw it in his face like that.
If he were stronger, he could've let her go. The Hunger Games were synonymous with the end to those who lived in the Districts. If you lost someone you loved to them, you weren't getting them back. He would've moved on and done what had to be done to keep their families alive and fed.
He was weak. He didn't want to leave District 12, the only place he'd ever known, so he let her disagree with him when he said they should run away. He didn't tell her he loved her; their bond ran too deep, and he wouldn't take a risk that might ruin it.
He was weak. So he refused to let her go.