fainjon asks two separate questions. The first is whether a square tip would give less drag than a rounded one. The second is whether the rounded tip causes the tip vortex to induce more drag. This second question seems to be referring specifically to induced drag, rather than overall drag, as the first question was. As has been pointed out, induced drag is usually a small component of overall drag. Induced drag becomes significant when the wing is operating at a high lift coefficient ( landing, slow speed flight, pylon turns, etc. ).

I think there is a widely held misconception about the cause of induced drag. The wingtip vortex itself is not the cause of induced drag, per se. The cause of induced drag is the distribution of downwash along the span of the wing. Downwash is the slight downward component of velocity of the air approaching the wing, caused by the very act of generating lift. You simply cannot prevent induced drag by making it more difficult for air to spill from the bottom of the wing to the top. To say that a square tip has lower drag for this reason is to misunderstand the phenomenon. It is possible that some wingtip designs could reduce induced drag by shifting the distribution of downwash ( usually this means that the tip design pushes the center of the vortex out from the end of the wing, emulating a longer wing ). Some winglet designs also take advantage of the tip vortex by using it to generate some thrust, which partially compensates for induced drag, but doesn't necessarily reduce it.

Since most of the time, induced drag is very small, it makes sense to consider other components of drag when choosing a wingtip. A square tip will certainly be more likely to have a region of local separation, causing additional drag ( profile drag ). Streamlining the wingtip is beneficial for the same reason that streamlining the rest of the airframe is beneficial. A rounded tip would certainly be closer to an ideal, streamlined shape than a square one.

It would be hard to quantitatively predict the relative overall drag between a square tip and a rounded one. My educated guess would be that the one which reduces the more important component of drag is better. This is particularly true since the other one cannot be expected to reduce any component of drag.

My advice is to not worry about induced drag or wingtip vortices, and round your wingtips. Experiments of the kind which have been suggested could shed additional light, if you care to run them.

banktoturn