As P factor is most prevalent in high angle of attack situations and pattern flying is about smooth/gentle flying, then I doubt P factor is the cause of your problem.
I had the same issue where I was applying rudder during pull ups. I found the stabs were not exactly aligned. I reset the stabs to perfect alignment and the problem disappeared.