That's true. If you were to move the pushrod all the way out on the servo arm, and all the way in on the control horn, it will take more power to keep the control surface in position. I wouldn't suggest doing that on full strip ailerons though, unless you're using high torque servos and 4-40 pushrods.
Another option is longer servo arms. You might have to turn the servo around or put the arm on the other side of the servo, but you will get your best torque on the control surface using the outermost hole on the control horn.
You never said what plane this was on. Something else to think about also ... the faster you go, the less control surface movement you'll need.