Harry Potter had always known he was as different from night and day compared to his relatives. His aunt was a busy body whom revaled in the fact that she was normal, but anyone whom new her knew she was not normal at all. The sames goes for Petunia's husband. He really tried to wear a cloak of normalcy and project that to everyone who knew him. But everyone knew he was a bully, overbearing sexist pig with anger issues. While his son took after the father while being horribally spoiled. Both him and his father made it look like they would decimate all whales through their size alone.
Although the family tried to wear the veneer of normalcy and nobility, they were seen for whom and what they were. What they happened to forget about is that while they mistreated him he would never forget. if they were dying he would cross the sidewalk and stay away from them.