Middle Earth was at war.
War is never easy, but unfortunately, in this one there was not one person who understood the situation. Truth be told, no one even knew which side they were on.
When two wizards clash, the most reasonable thing to do is get out of the way. If that is not an option, you better not be hasty in making the decision of who to back.
The power of both Saruman and Gandalf was known to all in Middle Earth. Saruman was the scholar, the wiseman who spent his days surrounded by ancient books. Gandalf was the adventurer who always talked with the Small Folk and meddled with matters that did not concern him.
One was good, one was bad - that much was clear to everyone. Saying which one was which... Well, that wasn't easy. And everyone knew, that if one was good and the other bad, then one had to be the Good and the other the Bad.
Saruman was siding with Sauron (people said that it was because of their similar names) and Gandalf was spreading rumor that Sauron was planning on world domination. He (or so Gandalf said) was going to achieve it with the help of the evil One Ring. Saruman objected. He claimed that the One Ring was not evil, at least not by definition. The inscription on it read:
One ring to rule them all,
One ring to find them,
One ring to bring them all ,
And in the darkness bind them.
While that may have seemed a bit dark and evil to the people, Saruman assured them that the ring was meant to help a worthy king get rid of misery and sickness. The unclear message of the inscription was apparently due to the dramatic nature of the people's ancestors.
While many people didn't quite believe Saruman (having been told stories of the evil One ring since they were children), very few of them stood by Gandalf. Many believed that the Gray/White wizard had gone mad. He had charged Erebor in an attempt to slaughter all the dwarves. No one knew the reasons for that spontaneous change of heart (this happened barely a year after the wizard helped them fight off a goblin army). Saruman had been heard saying that it's very easy for a wizard to lose a sense of self and go mad. Many people nodded their heads on hearing this and stated this was probably what had happened.
Still, many people refrained from joining Saruman. True, Gandalf had started gathering a army and was killing good people, but Saruman (along with Sauron) was the type of person that our mothers told us to beware of. Sitting for hours reading dusty old books and manuscripts - it just wasn't natural - people would scoff.
And so, the war went on. And still goes on to this day, as neither Gandalf nor Saruman managed to find strong allies. The people tend to ignore the two bickering wizards, as long as the two don't interfere with their normal, day-to-day lives. Sure, there a minor casualties from time to time, but death by wizard is deemed to be a natural death nowadays. That's what you get when you deal with wizards.