The P-factor only is active when the tail is still down and the prop is not perpendicular to the line of motion.
However, the engine torque will try to bank the plane left, and the prop wash will try to yaw the plane left. At higher speeds, these effects are getting smaller relative to the plane's normal stabilizing forces, but they are still there.
Prop sizes do make a difference, and it pays to experiment a bit. Props with much pitch ar the worst offenders.