Loyalty. Was that what they thought it was? Loyalty to Great Britain that kept him from joining his brother? He didn't think he looked or acted particularly devout or "loyal" to Arthur. But it didn't really matter to Matthew what they thought his motives were. Whether they thought it was hatred for Alfred or love for Arthur, or even just timidness.
They didn't know the truth, and they probably never would. How could they know of his devotion to his siblings, the other colonies? Arthur had barely paid them mind before the revolution, and he feared for them if they were left alone with a broke, depressed, and sulking post-revolutionary England. Matthew had never cared much for either independence or subjection and he didn't hold any particular fondness or dislike for either of his "older brothers". He supposed it was a sort of loyalty that kept him British, though it wasn't the loyalty that others assumed of him. No, Matthew's motives were his own and it didn't really matter what outsiders thought, as long as his siblings grew up better than him.