Alfred wanted to know Matthew more, he really did.

He also wanted to take him out to places publicly, but they were Countries, Political figures. But he did love him, so Matthew would remain his Dirty Little Secret.