I''m not sure what you are saying here.

That torque is used to push against the forces on the control surface. The larger the control surface and the faster you fly, the more force required to move the control surface away from center.

A servo has to work harder if you go farther out on the arm exactly for the reasons you describe. The servo has a fixed amount of torque. And moving the control arm farther out makes it harder for the servo to move the pushrod.

Along the same lines, moving the control linkage IN on the control surface is shortening the leverage the pushrod has against the control surface, meaning the servo again has to work harder.