Even knowing she had to leave eventually, it's different now that she's actually gone. It's not even like she's all that far away now, she's still in LA, they still see her so often it hardly qualifies as leaving, but she's not at the studio every day, she's not there at dress, she's not teasing them over flubbing a line, so she's gone in every way that matters.
Even when she wasn't around, Harry'd kept them both sane, and something like a family. Now Tom's never quite sure what to say anymore if it doesn't have to do with the show, and Simon's so much more likely to take some offhand comment and blow it all out of proportion, and there's an empty space where Harriet should be. It's almost always been the three of them, and Simon and Tom can't quite figure out how to be just two after that.