I had a SuperStroke Fatso 5.0 on a putter for a while, ridiculously big, but I wanted to try it. Made it annoying to pull in and out of the bag, but besides that it was fine. I don’t think anyone is doing themselves any sort of disservice with anything putting related because (as has been discussed in other threads) it is so incredibly personal and everyone changes over time, even the pros. These days, I’m playing a midsize pistol grip, not skinny and not massive, and it seems to be fine for me.
The only way to know if something different will work better is to try it out. If you have an air compressor, you can get the old oversized grip off pretty easily without damaging it and then can pop a different, smaller grip on it quick and give it a roll. If you don’t have an air compressor, most shops these days can regrip without damaging the old one, so always worth a shot if you want.