Harry wasn't so sure about staying with Mrs. Figg. She smelled like cats. Her home smelled like cats. The cats themselves were huge fluffy beasts. Some of the cats seemed even bigger than Harry!
Harry knew he wasn't a very large child. He was quite small, a freakish little shrimp. While useful in hiding, being small had many disadvantages. When cooking, he needed several stools to see the stove or to access the sink or to use the fridge or the oven. Gardening with huge adult size tools was also challenging. For Harry's little arms, the rake and shovels and various clippers were quite the struggle. Being small also meant Dudley and Uncle Vernon felt too large. They took up so much space and it made Harry nervous. Harry worried that one day there would no longer be space for anyone in the house besides Dudders and Vernon and Petunia.
It wasn't that Harry would mind taking a break from the Dursleys, it's just that whenever they left, Harry got stuck with Mrs. Figg. She wasn't very fun and seemed more strict than Petunia. If Harry ever had to babysit he'd do a better job. He'd make nice treats for the kids like jaffa cakes or even treacle tart! There would be happy colors around the house and lots of outside space to play and lots of books and games. Mrs. Figg had water and faded grey blue rooms and cats everywhere. A cat by itself was fine, but all together they were way too much.
Harry would prefer just being alone. He could take care of himself just fine thank you very much.