thought I'd retell what Ivory did to piss Max off so much in the first place.. The story is getting long and you may need a reminder..
1990 [well actually it's 2001 actually, but timelines, you know].
Turns out Max and his partner Alex had recently busted Ivory Smith for drug dealings, appearently Max had been annoying Ivory for quite some time, busted Ivory a number of times before as weell. But this time Ivory had enough. This time, after breaking out of jail, he needed to make sure Max stayed away from him, to send Max a message.
So one day after work Max comes home to find his house unusually quite. Ivory Smith manages to be the one who killed Max's wife and daughter, and in fact might actually be 'worse' than the in game killers, who at least had an excuse of being fucked up on drugs, but Ivory was not, his actions were completely intentional, and he made sure to leave messages to tell Max to keep away from him.
You can also here Ivory's last words to Michelle before Max found her, ''I'm gonna hurt you! I'm gonna hurt ya! It's coming. Death is coming. You're gonna die!''
Of coarse, Max does ''not'' stay away, he continues to be a pain up Ivory's ass, and eventually Ivory has some corrupt cops and frame Max while after Ivory kills Alex, and uses this as a chance to flee Jersey and head to Los Santos. Max later tracks and kills those corrupt cops, getting help from Dave Norton, leading to Max being involved in the prologue mission, as Michael ''claimed'' to know where Ivory was, but was revealed to be lying, which is why Max was so mad at Michael. That and he never met Trevor, and believed Michael was just being a poor friend.
All that would be bad enough on it's own, but there's also the fact that every evil and sadist action he does in Balor's stories are kept canon, he did all that other terrible stuff as well. It can even be implied he just did it cause Carly reminded him of Michelle.
So all in all, Ivory is always labeled as one of my ''most evil villains''. At least in the Max Payne crossover.