Jayne has never been vulnerable. He's never let down the walls of crudeness and violence that seem to define him. No one ever sees behind the lust of his hungry gaze to see that he's trying to find that one woman who'd gotten away, trying to replace her with everything not her.
But Zoe doesn't have to see it to know it's there. Loss always leaves its mark on a person, and even though she doesn't know what he lost, she knows he lost it. She wonders, sometimes, but in the end she knows it's not really important what it was, or who it was.
And even though he's nothing like the funny pilot and she's nothing like the delicate not-quite-society girl, they find ways to fill the holes a little, and that's all that matters.