This was written with the assumption that Gabriel Agreste is Hawk Moth.
Bearing a Miraculous comes with a price. The bearer has to offer their most treasured possession, be it physical or not, in return for the magical powers given to them.
For Marinette, her confidence around her significant other.
For Adrien, his relationships with his family.
For Gabriel, his humanity.
The metamorphosis of caterpillars into butterflies is a metaphor for change.
But change is not always good.
Every time he transforms into Hawk Moth, every time he sends out an akuma, he feels bits of his sanity slip away.
Gabriel Agreste had never expected that a simple butterfly brooch would lead to such chaos and calamity.
First, he had lost his wife.
Second, he had lost all feelings of attachment to his family. He had neglected his son, whom he knew heavily depended on him ever since his wife passed away, but no, because of the exchange, he no longer possesses the ability to love and care for his family.
Gabriel did not choose to wield the Miraculous. He was cursed the moment he touched it. He was tasked to steal other Miraculouses. Whoever corrupted the kwami living in the brooch had also corrupted the wielder. The kwami's original power was not to turn people into akumas but someone found out just how powerful a kwami's power could be and had exploited it. The dangerous entity currently lives inside the brooch and wants all the other Miraculouses.
Gabriel never wanted to harm anyone in Paris. He never wanted to harm Ladybug and Chat Noir. However, the entity living in the jewel has the ability to take over his body at certain times of the day to send out an akuma and he would wake up with gaps in his memory.
Gabriel was terrified. He knew he was dangerous to people. Dangerous to his only son, Adrien.
That was why he distanced himself.
To protect Adrien.