Magic unbeknownst to the major population magic is indeed real but the number of people able to use it is rare, not to the point where it's impossible for a person to never meet someone.
To put it simply Magic wielders have so little control over their magical abilities that they mistake their magical powers as an extension of their semblance.
For example, let's say that a person is capable of creating fire using their semblance, but for whatever reason, they suddenly gain the ability to create swords.
Most people would be excited and confused at first, but would most likely begin to use their fire ability and mix it with their newly gained ability to create swords.
Because using any other element becomes more and more difficult as the individual becomes more comfortable using a single element, they wither didn't bother trying or give up ever trying to use any other element.
The same goes for anything else that magic can be used for.
To put it simply, magic s something that depends on the individual, who has it.
Magic is also something that shouldn't be discredited as it has been known to keep its effect over long periods.
Weapons used by the empire are known to use magic to keep them from deteriorating and succumbing to the effects of time, however, even this form of magic depends on the strain it's put under, as a lot of the empire's technology is now lost.
There are said to be individuals capable of doing what one could consider miracles, but if those people exist they would be one in every million, or billion.
Ye despite what may seem as something useless or useful to you the reader, you must remember that if anyone were to ever encounter a person that uses magic, to not forget that magic can be stopped with either overwhelming numbers- firepower or a combination of both.