Rory had notice it first.
The way they gravitated towards each other subconsciously, at first but then as soon as one would realise they would slip apart only to come together again moments later. Flirty banter to hide the hurt and the pain. He doesn't know how it hasn't destroyed River already. The knowledge that the person that has been the main point all her life is forgetting her. Knowing her little by little, every time they meet the doctor knows her less and it is killing her.
Rory has seen the moments when she thinks no one is looking, the pain and sadness but acceptance that this has to happen no matter how she feels. She looks at the Doctor and even Amy with the same look of anguish and longing. Rory understands, sort of, he has been forgotten but thankfully it was quick he didn't have to watch Amy as she started to look through him, when she didn't trust him any more, when she turn to someone else for comfort or watch her look at others like that and not him, Rory was sure he would go mad, with what he wasn't sure. Sadness ? Grief ? Longing for something that can not be anymore?
But River is strong, the strongest person he knows, she tells the Doctor "no" that he's wrong, when he shouldn't. And the Doctor could be quite terrifying when he has to be, when he's angry, when the darkness starts to fill his eyes. He would stand over her like an angry god, glaring at her into submit to his whim, yet she stands up to him, her heart no doubt breaking because he doesn't know her or their relationship, he would snarl and snap and hiss half truths as if to hurt her. The Doctor guesses and knows she means something important to him in time but still in those mad moments he acts as if to break her, however you can't break what is already broken.
River will always be his child, she may be older in every way but she will always be his and Amy's lost Melody. Rory was happy to know his daughter would be happy because safe was never an option when running with the Doctor. You kept up or be lost behind to become his ghost.
But River and the Doctor are like a disaster waiting to happen, pulled together and torn apart only to meet again at different times, damaging themselves with the secretes of each other.
They are killing each other. Teasing each other, testing one another to see who can cause the most damage first, and nothing can soften the blows or try to stop it.
The Psychopath and her Mad Man.