(Disclaimer: Tolkien's, not mine.)
Part of what they said about him as a child was true - he was, indeed, scared of the horses. Horses are large and heavy and quick to panic, and he'd once seen a stable-boy kicked backwards over a trough, breaking his arm. It was sensible, not cowardly, to be wary of things that could be dangerous. And whatever else they called him, he wasn't stupid.
It is not possible to avoid horses in Rohan, and so he took great care to watch them. He lay on his stomach in the sun, or tucked fists deep inside his sleeves when the wind blew bitter and cold, and learnt how to read the language of snaked necks and flattened ears that horses use with each other. While he was doing this, he noticed something else: the stallions, all power and muscle and strength, never led the herd. It was always one of the older mares, usually bony and scarred and slow, that decided when they would go down to the stream or away to new grazing grounds. The stallions only ever followed.
He never wanted to command troops in battle. He knew where his strengths lay, and it wasn't in the world of spears and shields. He never wanted a crown, or to be carried high by a cheering crowd - such things were fickle, given just as easily to some grinning idiot with a good sword-arm as they were to those who earned them. In his youth, he still wanted their friendship, but the chance of that was kicked out of him by words and boots alike, and when it had gone he did not much miss it. He did, however, want to lead.
"A wise choice, my lord," he learnt to say. Or "Understandable, yes, but would this not be best instead?" In return, the people of Edoras spoke well of the wisdom of Theoden King. For many years he did not care what they thought of him - let them scorn him if they liked, let them mock him when he wasn't there. Theoden himself called him wise, trustworthy, and a loyal servant of his king. And so he went on.
No doubt he would have known better than to trust Saruman, in other circumstances. That day the wizard came to Edoras, he never suggested he should be permitted to stay longer than the brief reception Theoden allowed him. And no doubt it would have stayed that way, if Saruman hadn't thanked him personally afterwards. "Theoden King is fortunate to have such a wise companion to lead alongside him," he said. "The Rohirrim must respect you greatly." The tight feeling in Grima's throat reminded him of long ago.
A colleague, Saruman called him. An ally. The words were like honeycomb, too rich and warm to hold between his fingers. A friend. When all else crumbled, he still had these things to clutch at. He had never desired a great deal. This, this was all.
So when the halflings dared look at him with pity, and Saruman turned that laughing, twisted face towards him - You do what Sharkey says, always, don't you, Worm? Well, now he says: follow! - what else could he do?