Best speed for a given thrust is with the lowest drag. I don't care what the lift to drag ratio is if I got the required lift and the lowest drag. Maybe if I was flying to the maximum altitude it would become a much larger concern.

Flying unloaded (1g) you might make the argument for a short wing span, but once you have to turn to stay on a closed course, then wing span (or rather Span Loading) become much more important. Since you want the low drag in the turns as well as the straight-a-way. That why Formula One racers have increased the wing spans from the early days to the present. The main limitation is weight, since it take more material to build a long wing.

Aircraft design is a bit like rock-paper-scissors!