I found this headcannon in a fanfic and thought I would share it!
Dazai's Ability can be turned off. However, he commonly keeps it activated due to trust issues and because he unknowingly spent years with it being active. He only ever turns it off when he's either with someone he trusts completely or when he feels secure enough to let down his guard.
When Dazai was in the Port Mafia, the only time he ever turned it off was when he was with Chuuya Nakahara. Because of the nature behind Dazai's "passive" gift to nullify other Abilities, no one really noticed when he turned it off around Chuuya during missions when he was sure that there wouldn't be another Ability user.
Not even Oda was trusted enough by Dazai to deactivate his ability once he cultivated a relationship with him. Just Chuuya.
.
After Oda's death, Dazai left the Port Mafia without saying much to anyone. He did leave Chuuya a little note to apologize for leaving him and explaining his reasons without going too in depth of them. He also repeatedly assure the ginger that his choice to leave was not at all influenced by anything Chuuya did or said or not do or say. (He didn't apologize for blowing up his car though. But Dazai did leave him a small gift!)
When Dazai joined the Armed Detective Agency, before Atsushi left his orphanage, Dazai did try to show some faith in front of Ranpo and Fukuzawa and turned off his gift once for Yosano to use hers on him when she complained about his poor health condition after he passed his entrance exam. He also justified it to himself by claiming to hate pain and wanting to be physically healed.
Because he received treatment from Yosano's Ability on him, all of his physical scars were erased but he kept wearing a lot of bandages as armor and a barrier to hide his self behind. (Not literally, of course, but sometimes, he couldn't help but do it literally too. I'll put up another chapter for that particular Headcannon.) After that single show of tentative trust, Ranpo had allowed Dazai to take the exam freely with Kunikida being the one to test him instead.
.
Dazai didn't deactivate his gift at all again until he reunited with Chuuya in one of the Port Mafia's basements. Again, Chuuya didn't notice Dazai's Ability being quiet because he was too pissed and more focused on beating up Dazai.
It wasn't until Dazai made an honest attempt at his life by sneaking through one of the Port Mafia towers and letting himself fall off just as Atsushi burst onto the rooftop with Kunikida and Akutagawa that Chuuya finally noticed Dazai turning off No Longer Human. Chuuya hadn't been on the roof but had been already about halfway up the building and broke through one of the windows when he saw Dazai falling and had instantly tried to use his Ability on him. To his surprise, For The Tainted Sorrow had not been nullified even though he used it on a well-known nullifier.