Based on the description I'd say the control rod is flexing and the pressure on the control surface is enough to bend it slightly. Add some guide supports along the length of the rod or change to a stiffer one.

Try moving the elevator by hand while looking at the rod inside the plane to see if it does flex (don't use more than one or two lbs of force). Keep an eye on the control horn. too. Some are "flexi". Could also be slop in the servo to control to horn linkages. Should be none. Ony takes 1/16" of control movement to make a noticable wobble.