I run a 2 blade prop, have no throttle to rudder mix and the plane goes up straight and doesn't veer off during pulls or pushes.

As I understand it, (correct me if I have it wrong), P factor is caused by a high angle of attack attitude, so maybe try a more gentle pull or push?