He was always the different one. The one that nobody bought presents for at Christmas, or on his birthday because he didn't have enough friends, because he was different, because he scared them.
He took their stuff, but nobody knew where it had gone. He tortured their friends, but nobody knew how he did it. How could an eight year old boy permanently paralyse their thirty four year old carer? It just wasn't natural and he just wasn't normal.
But, did any of them ever stop to think what it was like for him, to receive no presents, to think that nobody cared. Did they ever stop to think that it hurt him, the names, being freak boy and Riddling Riddle, being scary without actually meaning to be? He didn't mean to hurt Mrs Donald, he was just so, so angry, and then all of a sudden she was lying on the floor and she couldn't move and he didn't know what to do.
Of course, later on he knows exactly what it is he's doing, he enjoys it almost, torturing people whilst they beg for mercy, for death, for anything but this.
Do they ever think, the Golden Trio, Dumbledore, the Order; do they ever stop to wonder what it was that made him like this? Of course they don't. He is the enemy, ruthless, heartless and evil, and must be destroyed, no matter what the cost. They never stop to think that before the cruel shell of a man, came a scared little boy.
Because there is a means to every end.