As the closeness between Matsuda and L increased, so did the touching. And, as with the closeness, it was L who initiated it. It wasn't like they were groping each other or doing some really intimate touching - just simple little brushes that originally made Matsuda flustered and on edge and now only made him flustered (and even then, not as much). Casually, L would keep his hand against Matsuda's a moment longer than necessary. He'd brush his hand against Matsuda's back or hair, depending on whether Matsuda was sitting or standing. It wasn't so much that L was flirting; he was merely curious.
And as the contact between Matsuda and L grew, so did L's curiosity. He gradually went from small touches to holding Matsuda's hand or grabbing onto his sleeve. Like an elderly person, it seemed that L really enjoyed being able to hold someone's hand - Matsuda's especially. Not that he held the other members hands; that'd be gross. No, with Matsuda, it was like L had found someone he could share his love of sweets with. Although he wouldn't offer Matsuda any cake, he didn't say Matsuda couldn't have any cake if the younger detective asked.
To all the members (except Matsuda, of course), it was obvious that L was slowly gaining a crush on Matsuda Tarou. And while it amused Ukita and Souchirou (who knew what Mogi thought), it annoyed Aizawa. How was he going to get any work done if the leader of the group and the idiot of the group got all lovey-dovey?!