That's true for any "two dimensional" maneuver ; a loop, roll, etc. If one side is heavier you'll drift sideways.

There is also a torque from the prop that pulls you to the left, so some right thrust is usually built into a design. There is an optimum amount of right and down thrust for any speed, so that means you have to compensate for other speeds above of below that ideal one.

So, practice with the plane going away from you. Then practice with the plane passing in front of you but going vertical at the closest point. Then combine what you learned to keep it straight on both axis.

And, just as in the full scale world, some planes just aren't designed or powered for great vertical performance. And being able to compensate for wind direction and the planes attitude to make it fly crisp and steady takes a LOT of technique, which only comes with practice. Watch a pattern model do a diamond loop at the same speed on all four legs. There's a lot that goes into pulling off a maneuver that appears pretty simple.