There still is a lot of force involved. Even though the servo is only moving the swashplate, the forces are still transmitted from the rotor blades, through the arms, and back to the servos. There are no free rides.

You are right, its not like moving a big elevator 45 degrees. But any helicopter pilot will tell you, use good strong high torque servos on the controls if you are going to be flipping around your helicopter in 3D flight. The forces are there, and good tight linkages need to be employed to create a strong precision linkage.