Check the play in your main wheels. They should both spin freely. If the right wheel is much tighter than the left, it act as described.

Nevermind, I misread the question and thought you said as soon as they start rolling.