He had never actually gotten his hands dirty. It was easy enough to get other people to do the dirty work for him. There was the one time he was going to take matters into his own hands, and what had it gotten him? A shattered sword, and a lifetime of wandering in exile.
When he was first exiled to Elba, he had met a man who told him in great detail about how he could choke someone with his own hands. That was going to be a line he wouldn't cross. Having power meant keeping his own hands clean.