If P-factor is so important and torque is not why does the same thing happen to dragsters?  Tricycle gear are less susceptible because the nose wheel tends to stops the left movement.  And pattern planes typically have large, flat verticle surfaces that take out the spiralling airflow quite a bit.

Try a full acceleration from stop with a taildragger and you will see it spin around in a circle - and there is NO P-factor until the plane starts moving.  That's torque!