what I have always done is used the pointed base of a metal file. I start very gently, and turn it both ways, making sure I go all the way around so I dont make an oval. They have a very shallow taper, so its easy to take out just the right amount.

Also, I do this Before I balance the prop, so if I get off center with the hole it will show up then.