For what it's worth, I was getting that same big swing side to side. I found out that it was due to my stick inputs. Give the heli a nudge to correct / level the flight, then immediately center the stick and watch for the flybar action. When I started flying using a series of "Pokes" at the stick and immediately letting it center, I started getting a lot better quickly.

You probably know that as you have posed video of you flying pretty well, but others just starting might find it helpful.