Mr. and Mrs. Dursley stared at the basket that was sitting on their kitchen table, in the middle of the perfect kitchen Mrs. Dursley worked so hard to maintain.
"He's one of them, Pet. I won't have it."
Mrs. Dursley nodded dumbly, unable to think of anything other than her shattered dreams. Dudley, her perfect son – she had sworn he wouldn't have a sibling abandon him, the way Lily had abandoned her. She would give anything not to have her boy go through that. She'd even give Lily's son to someone else – but Dumbledore's letter had made clear that wasn't an option. The boy would be killed anywhere else.
Mr. Dursley's mind was more focused. Magic – it was unnatural, a cheat. Work was meant to be done with one's hands, through sweat and toil, not by waving some stick! He wouldn't have his son growing up thinking you could take shortcuts in life.
"I won't have it, Petunia. I won't have our son believing in freakishness."
"We have to keep the boy. We've no choice."
"We'll stomp it out of him, then. Make sure he understands the value of hard work."
Mrs. Dursley glanced at the letter again, then at the sleeping baby. "We'll put him with Dudley. It'll be easier to take care of them together."
Mr. Dursley hesitated before agreeing. "Fine. The moment he shows signs, though, he's out! I'm bringing my boy up honestly!"
"Yes, of course." Privately, Mrs. Dursley was making plans to keep the boys from becoming too close. Better for Dudley to lose a playmate than a brother, when the time came.
Together, the Dursleys made plans for the future. It was cruel, but in the couple's eyes, they had no other real options. In the end, it was all for the greater good.
End Notes: I find it really hard to believe that anyone could just throw a baby into a closet and be done with the matter. I think it's very likely Harry and Dudley were given similar care for the first few years, until Harry first showed signs of accidental magic. At that point, he was moved to the cupboard, and given chores, to try and "teach" him how to work hard.