Normally, the rudder is the first place you need a stronger servo.

The ailerons tend to get the extra power needed by getting the dual servo treatment.

The rudder has to deal with the tailwheel (or nosewheel) loads... and its often got more area than the elevator. Rudder has always been the surface I've had the most trouble with preventing flutter on when using standard servos... and I tend to put a 70 in-oz there while still using a S-3004 on elevator in a .60 size model.