The control horn on the servo could be slipping under load.
The control system could be too loose... increasing speed adds download to the horizontal. If the elevator can move down because of a loose system, it will, and the plane will dive.
The same thing going up..might be a loose servo mount itself.