I can tell ya that for each and every plane I assemble, the worse part is installing the control horns. That's mainly because I, just like you, and obviously others..

have a tough time getting those darned things lined up correctly, and end up with a hole that is much larger than I had anticipated. And Minnflyer is right.. all you have to do is clamp them down right and they work fine.
I've seen people drill out a 1 inch hole, then fit in a hardwood dowl cut to match then sanded down to match the taper of the aileron, rudder, or elevator, then drill through that. They call it a hard point. I've never done it. I use the CA in the hole method to harden up what's underneath the control horn mounts. And from what I've been reading it's probably not necessary. But I still do it. Can't hurt, I guess, because it does harden things up a tad under these things. The added CA will fuel proof the bare wood if you should happen to tear the covering under the hinge. At least that's what I think happens. Maybe or maybe not... but I still do it.
CGr.