What airplane are you using? Is it in a 45deg upline or downline? or horizontal?

If it's a good (and gotta be well trimmed) 2M airplane and it's in the 45's it should just about be fine with aileron only (maybe just a slight adjustment 'squeeze' here and there ). If you are adding rudder/elevator out of habit, it could be throwing you off? If it's in the horizontal, yeah could be a "feel"/timing issue .. or maybe badly trimmed? (just thinking).

Otherwise, practice and get a pattern friend to note when it's happening and reference what you might be doing at that point ?? . . and then practice some more

JB