RE: Can you ruin a servo by......
Brian,
No damage if you grab the arm of the servo.
If you try doing it by grabbing the control surface that the servo serves, you can bend the connecting rods, loosen anchoring screws of the horns or servo, damage the hinges or break nylon clevises.
That is one of the reasons why full scale airplanes have the control surfaces locked, so they cannot play with the wind and damage the control mechanisms (no servos there, though).