The memory of Milah stays longer than the woman herself ever did (poor, disappointed Milah, yearning for adventure - Milah, the lost - that heartless whore) but its sting eventually lessens and fades.
When he meets Cora, her child already links their futures; but Rumpelstiltskin seeks to bind them tighter, to sweep her away into a grand adventure and, once denied, wonders when he'll forget her as well.
Then Belle is living in his castle, dreaming up her own adventure within its walls, claiming to dream of him even as he sends her away (to her death) and this he'll forever regret.