Disclaimer: I don't own Merlin.
Merlin was no stranger to danger. Ealdor had its own dangers that he had had to deal with while growing up. Food shortages were the biggest dangers. Starvation was not something that was uncommon. Winters brought the danger of freezing to death. However, Merlin had never given much thought to these things before. They were just simple facts of life that he had grown up with.
When he moved to Camelot, he was faced with a whole new kind of danger. Camelot was constantly being attacked by some form of creature or vengeful person. Going out into the forest was dangerous because of the bandits that lurked within. However, those were not the most dangerous things he faced.
Uther's hatred of magic was the biggest danger. His hatred meant that what made up Merlin's being, his magic, could cost him his life if it was ever discovered. This one danger, more than all the other dangers combined, scared him more than anything.