hmm, It could be dodgy bearings on one side, and when the rear drive is on its just over come by having 4 driven wheels.

Or it could be some wierd diff issue.

or slitch less likely it could just be the roataing mass of the centre diff. but the mainly effects touring cars i think.

What side does it drift to?

Are all the bearings good?