Check to see if there is any warp in one, or both, wings, tail, rudder. It sounds like you are able to trim it initially, but airspeed changes are producing more or less lift in one direction. The same concept as a plane rolling faster when it's going faster.

Just a thought.
Jet