It's the looks that get to him the most. The ones that attempt to say "I care nothing for you" while still implying "you are mine, nobody else may touch." He's never quite certain how to react.
In his mind, Subaru knows that he is merely a toy for the Sakurazukamori, a game to play, for a time. Seishirou has a bad habit of breaking his toys. But still, the contradictions in that look confuse him. And, perversely, amuses him. He supposes it's natural, after so many years of being owned by Seishirou, that he too would be able to find amusement in other's emotional turmoil.
Because in his heart, Subaru knows that the other man loves him. A sick, obsessive love, but emotion nonetheless. Subaru knows that he's still dangerous, and it's a good thing the Sakurazukamori has never taken offense at his smiles, because he doubts he'd be able to stop.
Love will never keep Seishirou from killing. Especially this, a love that Seishirou refuses to admit to, and that denial is what makes Subaru smile. Seishirou loves that smile best of all (but without ever telling), because when Subaru smiles like that, he's smiling just like Seishirou.