It killed him to not be able to be at Frodo's side, when he wanted to be there so very much.
When Gandalf had rescued him and his master from the slope of Mount Doom, Sam had been overjoyed, but far too exhausted to show any of his jubilation over the wizard being alive. They'd been whisked off to Minas Tirith, and for only the second time since the Fellowship had set out from Rivendell, he and Frodo had been separated.
If it was possible for Sam to speed along his recovery just so he could be at Frodo's side, he had done so.
Now, standing in the doorway to his master's room, watching the remnants of the Fellowship of Nine listen to Merry and Pippin talk animatedly, Sam shared a secret smile with Frodo. For though all of them had been effected in some way by the war raged over the One Ring, none of them besides himself, Frodo, and perhaps Aragorn, truly knew the cost of carrying the Ring itself. Sam had seen firsthand what the Ring had done to his beloved Mister Frodo, and he knew that image would never leave him. To the end of his days he would remember the kind of power that the Ring could provide came at a price.
And I'm so glad you didn't have to pay that price, Mister Frodo. Not entirely, at least.