Right!

And that's the problem... that you need to drill perpendicular to the surface chord line, which is at an angle to the surface itself.

A drill press and a jig makes this much easier but it is a lot to deal with for a small hole.

I end up using a small finger drill, will looking at the control surface from the side, edge-on.