After soloboss's post for the hell of it I took the rotors off boilded them and then checked the balance. Sure enough the top rotors were way off. Basically I had to put tape all down the underside of one of the blades to compensate. I can't believe it but that was definitely the majority of my problem. It still will only hover in place with only left stick movement for about 4 or 5 seconds before I need to use the right stick a bit but it is WAY better than before. I'm guessing it's likely that if I go back and redo my previous tweaking steps, flybar adjustment etc, that I will be able to get it even more stable.

That blows me away. I mean out of the box it was completely unusable and the main reason would seem to have been the condition of the rotors. That's pretty nasty. I'm guessing a lot of people would have just tried to return the heli or just put it in a closet somewhere and forgotten about it...