If he could, Cullen would have refused Lavellan from going into Adamant - at least without him by her side.
He knows Bull will fight with and by her, and he's seen what Hawke is capable of. Stroud is clearly capable as well, for having survived this long.
Cole wants to help others, but he's a spirit and Creatures of the fade are not to be trusted. He doesn't know what to think about Solas; hasn't cared to get to know the mage.
But he puts his trust in the men by her side to keep her safe and to bring her back to him.
Because he has to for everyone's sake; not just for his own sanity but for the sake of all of Thedas.
But then the dragon shows up and then she's GONE and Cullen is out of his mind with fear.
He can't stay still, but paces around and won't hear a single word of consolation ("She walked out of the Fade once before, Cullen, she'll come back this time as well." Cassandra tells him, but how can she know?).
He doesn't know how long she's gone (yes he does, down to the last second, though he'll never tell anyone that, not even her), but when she finally returns he has to restrain himself from dragging her back to Skyhold himself, to stop himself from taking her away from the danger the Grey Wardens possess.
His nightmares about watching his former friends and comrades being turned into abominations now turn into haunting nightmares about Lavellan becoming one and how he alone must be the one to kill what she has become.
What makes it worse is that he doesn't know if he could do it if it ever came to that.