You'd have to dig hard to find a definitive answer to "what is the right amount of torque".
The HS55 is an example of a servo with sufficient torque but a crappy gear train. The HS65HB has proven to be more than adequate for any 1/2A aircraft so is a go-to choice for many. I agree that the gear train and vibration resistance are bigger issues. I have yet to see an example of a 1/2A that stalls the elevator servo in flight, unless it is due to stiff hinges and/or crappy linkages.
You probably need less torque than you think.. you can fly a 170mph .50 delta with elevons on 48 oz-in servos. But it is generally not recommended..