I have used large diameter end mills with four or more flutes. Chuck them up into a drill press, run as high RPM (2000+) as possible before tounching down to cut a hole. Sharp ones can be found at swap meets around here for about $2 each. Most often they are very accurate as to diameter, unless they have had their sides sharpened along the way. Other problem may be in that for a tight fit, your wooden dowels may not be all that accurate of a diameter.

Wm.