The only thing I will add to Dick's comments is hat if the horn is too large, you can experience "snatch".

The term comes from full size aircraft when, with too large a horn on the control, a slight deflection can result in the horn moving the rest of the control surface, thus "snatching" the stick out of the pilot's hands.

The servo is not going to let go of the control surface, but too large a horn may cause a twist in the elevator (in either direction) which may be the reason for what you describe.