When Hans said his brothers had pretended he didn't exist for years, he wasn't exagerrating. Being the youngest of thirteen put him in the mostly unlucky of spots. There was no chance he would ever be king, and all of his brothers had practice torturing and being tortured by each other. In that sense he "benefitted," from their advanced skills.
Those three years of isolation had driven him mad. What Hans hadn't told Anna was that not only did his brothers ignore him, they had instructed the staff to do the same. They would serve him food at meal times, dress and clean him, but they were not to speak a word or acknowledge his presence past their mandatory duties.
The question then becomes what were his parents doing during all of this abuse? Well, the King was busy dealing with the economic times and how Arendelle had suddenly reduced its trade ties. The queen was busy playing the part of the devoted wife while drinking away her boredom and stress in private. The only time the parents saw their children was at dinner, and even then Hans was the youngest and with so much noise from all the other boys they could barely hear him, let alone help.
After those three years he had put on a friendly smile and had exacted his revenge on his brothers bit by bit. He was clever, he had a good poker face, and they all thought he was gullible from how easily he seemed to forgive them. Of course Hans never did, but he had also learned the value of only trusting himself.
Isolation had many Anna naïve and gullible, while it had made Hans untrusting and wounded. The only thing they had in common was that they noticed each other. Hans was often mistaken for his brothers, and Anna could only chat to a certain point with the castle staff, both were left alone and unrecognized until the moment Hans' horse bumped into her. She saw him as the first man she could talk to, and he saw her as gullible prey.
That wasn't all he saw though. He could see how she awkwardly conversed, how she enjoyed the party but had no idea how to act, and how her sister seemed more like a stranger. He saw all the signs of isolation and couldn't help but wonder why she was so different than him. Why she could laugh so freely, trust so easily? He had everything to gain from a marriage to her, but what could she get in return? But, she was sweet. Her laugh was beautiful and just a little bit, Hans let down his guard. He didn't love her, but he could see himself being less lonely with her, enjoying her laughter and smiles and kiss.
Of course it had come to him as a shock when Elsa's secret was revealed, but he decided to stay and be supportive. It was the character he was playing afterall. Left on his own, he saw how well fitted he was to rule. It wasn't just the power he had, it was the respect of people, everyone saw him. Even if it was just a title, people were talking to him, asking for his advice and council, allowing him to have an impact.
Then he saw his horse return and Anna was gone. He could have sent a search party and stayed behind, he could have gone on his own and pretended she had died or do the deed himself if he found her. Instead, he went with a large group after her. He had to save her. Part of his head told him he was being stupid, another part saying it was a wise business decision, and yet a third whispering into his ear that he might actually have a heart and Anna was in it.
He'd come back with her sister. In a way, he liked Elsa. He didn't like her personality, but he liked the position she had put herself in, allowing him to get what he finally wanted. He still had to play his role though… and he couldn't bring himself to be cruel to someone who hadn't done anything wrong. He could see the isolation in her as well.
Then though, everything broke. Anna needed a kiss of true love. Hans didn't love her, he liked her, but he had no respect for her. At best she was a pet to him, and she would die. In one moment he saw all of his hard work going away along with the one person who could actually make him smile. He broke. All of his inner evil, those little thoughts he'd hidden away for years and years, came out and began pulling the strings. Even then though, he couldn't kill Anna directly. He left her to die, but there was a small fire in his heart wondering if someone else could save her. All he had left to do was get rid of Elsa.
That was another snag. Bit by bit everything he had worked for was falling apart. The only thing he had left was the possibility of ruling, and nothing was going to take that away from him. He already had the unlucky number thirteen over his head, if people heard he had gone to a kingdom and both princesses had died he would never have another chance. His only option was to eliminate Elsa. But that didn't work.
Hans sat on the boat, in his little cell on his way to the Southern Isles. Hans was twisted to begin with, but Arendelle broke him. Sitting in his cell, he no longer talked to try and win people over. He just sat there quietly, the water rocking the boat and he could only think of how everything could have been. He could have been king, he could have been somebody, and with her, he might have been happy.