Depends on the plane and the type of flying your are doing, you MIGHT be able to get by with a lower torque servo. The biggest issue you are going to face will be that the servo won't be able to handle the load put on it. The results of that could range from not being able to move the control surface adequately while flying to the servo actually being damaged or stripped out. Either one of those could easily result in a crashed plane. With high torque servos prices coming down so much lately you can get good higher torque servos for not much money.  Personally I wouldn't risk losing a plane over a few dollars in servos.

What servos are you consdering using??

Ken