I think most recreational golfers should work to improve their full swing. Better swings will mean more accuracy, more consistency, AND more distance. And as you say, more distance off the tee means shorter irons to the green, but better swing mechanics will also mean that more of those short irons will be from the fairway.
I’d take issue with “invariably”. I know a bunch of guys who can really move it, and most of them are straighter than I am too. I do know a few wild ones, but most guys with good mechanics are both long and straight.