Yup the biggest problem is getting the holes aligned because you are effectively drilling at an angle.

I've had much better luck since I started using a drill press, with a wooden shim under the wing, etc., which is to be drilled.

Of course this requires that you drill the holes BEFORE you start assembling your ARF, but usually this is easy to do.

I've taken to starting with the motor and cowl, then control horns, before moving on to the rest of the plane.