Is flying a heli backwards really hard or do I just have some kind of mental coordination difficulty with it?

I practice it everyday on the sim, but I still have some crashes there. And in the real world, every time I try it's a near disaster - and I get in trouble with it before I even attempt a turn. I can bring the heli into me with the tail toward me from way out there, but when I try flying left to right or right to left, I have trouble.

I'm hovering inverted and even doing easy circuits inverted, but backwards flying seems impossible.

Was wondering if others had a lot of trouble with backwards and maybe someone has some tips.

Raptor 50
Hyper 50
GY401