You are absolutely correct in your analysis of differential thrust across the propeller disk. There is in fact a left turning moment applied to the propeller due to the asymmetric thrust distribution. The fact that is being ignored is that the rotating propeller is a gyroscope. A force applied normal to the plane of rotation of a gyroscope appears ninety degrees out of phase at the hub. Therefore the left turning moment on the propeller disk affects the airplane as a pitching moment rather than a yawing moment.

Thus when an airplane is pitched upward, the P-factor tends to pitch it up further. This is one of several factors that cause a typical airplane of conventional configuration to be less stable power on than power off.

This gyroscopic effect is particularly noticeable in case of a helicopter, where control inputs provide differential force on the rotor ninety degrees ahead of the required maneuver.