It is likely that Dumbledore hadn't noticed the warning signs, or if he did merely hadn't cared considering the fact that the man hadn't been averse to sending a child back into an abusive situation more than fifty years later.

There were several things which Harry who hadn't been raised in a normal environment himself didn't bother to question when he followed Dumbledore's "Tom was born evil" train when Dumbledore had shown him his memories of Tom Riddle during his sixth year. One of them was "How the hell did an orphanage whose only abundant resource was children manage to afford trips to the seaside?".

The answer to that question my dears was one not so noble benefactor.

Tom Marvolo Riddle had made his first kill at quite a young age. Tom's first kill hadn't been made alone, it had been made with a pair of accomplices who were never the same after that long ago day in a cave by the seaside, Dennis Bishop and Amy Benson.

The elderly man who had paid for Wool's Orphanage's trip to the seaside was wealthy, and regarded by the locals as a good and generous man. That man had a dark secret however, that secret was that he knew full well what his son's preferences were, and what his son got up to when he wasn't around. The old man also knew that by bringing the little orphans out to his country estate for the weekend, he'd just given his son a veritable smorgasbord to choose from.

Choose his son did, and all too soon, three rather attractive children vanished from the main group. Hours later, all three children returned without the man's son. The three of them were freezing cold, and soaking wet as if they had gone swimming with all of their clothes on. Two were sobbing and refused to speak, and the third...the eyes of the third were hard and cold.

The old man's son was never seen again after that day. There had been rumors that he'd run off. The old man knew however, the old man knew that had he chosen to search a certain cave that was only accessible at low tide, there was a good chance that he'd find his son.