He knew from the very beginning that loving him was wrong, not because the man in the military coat didn't deserve being loved, because he had realized that it would be unrequited. This was just sex, two bodies sharing and enjoying attraction.
But when the Welshman realized that for him it was so much more, he panicked and his shyness and invisibility returned. He swore that "his" Captain would never know about his change of heart; he would go on being his part time shag for the others while, deep down inside his heart, he would love him in silence.