There are various reasons for that to happen. First be sure its not a mechanical issue where there is a pushrod flexing when its pushing. Take a look and see if the model is weak in the direction that the rudder is pushing as opposed to pulling.
If thats not an issue, be sure you have equal rudder and aileron throw left and right.
If thats not the case, start to look for servo blowback as a culprit. See if the model snaps more similar left and right if its at low speeds. Check that by snapping power off on vertical down lines.

If all of that proves to be ok, compensate by changing the rudder and/or aileron throw in the weak direction to make the snaps look the same.