If the reamer isn't reaming the material up to size, it's dull. A dull edge tends not to cut but skates down the hole, especially with soft materials like plastics, brass, bronze. That's why you're getting a small (tight) hole. The material may be tough, but it's being pushed out of the way rather than being cut. The reamer must be DEAD SHARP. The lead-in (the angled cutting edge at the start of the reamer) does the bulk of the cutting with the flute's cutting edges scraping out the hole and finishing the job. I've never had any experience with smaller sizes of CORE DRILLS, or if the even make them in the smaller size, but this would be another avenue to explore. A core drill has more than two flutes and is designed to follow an existing hole, and does the job nicely. You can find core drills and quality reamers at an INDUSTRIAL cutter shop. A reasonable supplier online is Enco (www.use-enco.com). If you don't buy the cheapest thing they have, I think you'll have good luck with their products.
It's been said (and rightly so) that the most dangerous machine in a machine shop is the DRILL PRESS. That's because it's the least respected. I've personally seen a person's finger PULLED OFF by a drill press. He was holding a part in cloth shop towel when somehow, it became entangled in the drill, and off came the finger wrapped in the shop towel. The machine didn't even hesitate while pulling the finger out of it's socket and off! You can take the advice of some and wear gloves to hold props down on a drill press bed, but I really wouldn't recommend it.