Tony really didn't play well with others.
With other kids, he'd usually broken their toys, or pulled them apart trying to 'improve' them. No one wanted to play the rough and fast games Tony did. The tears were usually his fault. Tony got sent to detention a lot, top grades regardless.
Adults didn't generally like him much either, at least when they got to know him. Tony's 'friends' were his fans and followers. Still no one wanted to play with Tony.
But then Thor had given as good as he got. He couldn't break Steve's toys (what was that shield made of, anyway?!). Natasha always seemed to know what he was up to. There was no chance he could push Bruce around. Clint was better than him at a lot of things. Showing off just didn't work with these guys.
Maybe Tony didn't play well with others. But these guys weren't playing around.