Mal respected Simon. That would be a surprise to most, the two men argued and even fought from time to time. Mal respected Simon because he knew the man would do anything to protect his little sister. Mal would do anything to protect his crew so he could hardly fault Simon for thinking the same about his own flesh and blood. But that was also the reason Mal didn't trust Simon one inch.
Kayle had accused Mal of keeping Simon and River at a distance, of not letting them be part of the family that was Serenity, that was true. See Mal knew that if he had to Simon would let them any of them die to protect River, he had nearly done it the day Kaylee was shot. It had been the same when the Reavers had chased them after the bank job. Simon had been furious that River had been in danger, not even caring that if she hadn't been there, hadn't felt the Reavers coming, then Mal, Zoe and Jayne would all have died.
Mal also knew that one day Simon's determination to protect River at all costs would come head to head with Mal's will to protect the crew and, as Jayne had once said, that would be an interesting day.