He wasn't the type to keep secrets from his wife, he told her everything there was to know about him. Sasuke had even told Sakura the truth about the massacre, he was both angry and relieved that Kakashi and Naruto had not told Sakura the truth, she deserved to hear it from him.
That secret, however, was revealed years ago, before she was even his wife. The secret he kept from her now was his shame, his weakness. It was not until Karin had left Sakura and him alone with their newborn daughter that he actually revealed it. He didn't know what it was that took his mind to that night, maybe it was seeing Sakura with Sarada in her arms, he was in awe of the sight before him, in love with it. It was like his heart was bursting, he didn't know he could feel a love so pure, it was with this feeling that he let it slip. "I am glad I didn't take you."
Sakura's attention had been so completely on Sarada that she looked at him completely perplexed with a silent plea for him to explain. He had not meant to say the words, to confess his moment of weakness of so long ago, but now that he had said those words, it was impossible to bring them back. "That night, when you tried to stop me," he paused, they had never spoken of the night, they acknowledged its existence through other means, but they never spoke of it. He was going to break their unwritten rule. "That night, when I left Konoha, when you said you would come with me, there was a part of me that has always regretted not taking you with me. That night, that offer, it was so tempting, I almost said yes." He was looking into her eyes, waiting for a reaction besides surprise, when he could read nothing else he continued. "Up until this moment, if I could go back in time I would have taken you with me."
That was his shame, his weakness, how easily he would ruin her life like that. This selfish desire to be with her had been there before he could even properly acknowledge his feelings. To some it may not seem like a horrible secret, but to him it was, the simple notion that if he could do it all over again, the crash and the burn, he for so long thought that taking her with him would somehow be better. It wasn't, he knew that. It was wrong to drag her down with him, he was glad there had been enough sense in him at the time to leave her behind that night. He approached her slowly, sat down next to her, there was an apology in the way he looked at her.
Her eyes left him and went back to the infant in her arms, there were unshed tears in her eyes, she looked to Sarada but spoke to him, "There is nothing to forgive." There were more words in the silence, stories that only they could understand, and a love so deep for the daughter she held in her arms.