So there's a couple of things I'd like to clarify about Cosmo's childhood.
Linnie was gone for a good 3k years but definitely not by choice. When he and Nora were teaching Cosmo how to use a training wand, he was too enthusiastic with the wand. Although the magic was initially targeting Nora, she pulled Linnie in front of her so that he'd turn into a fly instead.
Linnie felt his conscious coming in and out at the same time, alongside not having his glasses, making watching where he was going hard. Every chance he'd get however, he'd still buzz around the house (both in hopes that someone would turn him back or that his kids would know that he still loves them)
He also had to avoid running into Nora as well, because he knew that she would try to kill them if he was spotted. So Linnie never actually "left" the kids...the kids just perceive it that way, and of course, Linnie could never forgive himself.
As for Chelsea, she was a bit more focused on keeping herself safe rather than what Darren was doing. As a matter of fact, because she knew that he was stealing from people, she disapproved of him greatly and feared that she was seeing Nora inside him. She felt that there were more honest ways for Darren to earn money for his siblings. Chelsea has a lot of trust issues when it comes to Darren.