Downforcerecon,

Ok, 99.9 percent of the time it is the pilot. I've had my AXE CP for a little over 3 months and I've only been able to hold the heli in a flat motionless hover a few times and each time it only lasted 30 to 45 seconds. That tells me that it is possible, but requires a lot more practice. I've been flying inside a KC 135 hangar and even in there there are breezes and drafts. I'd imagine there are also breezes or drafts where you are flying as well and that will bump and move your heli.
I've been working on stable controled hovers. No, that doesn't mean a perfectly flat not going anywhere hovers. It's in the air, a few feet in front of me. I've also been working on nose and tail in hovers. Once I'm able to keep it inside a 6 foot circle, then I'll spend a few minutes on that perfectly flat, not going anywhere hover, then maybe some forward flight.
So in answer to your question. It takes practice to get that kind of hover. When you're hovering, how high above the ground are you? You need to be between 3 and 5 feet to get out of the heli's blade wash (turbulence created by the main rotor blades). Blade wash also has an adverse effect on your ability to hover.
The short skinny of is it takes time and practice.
Sorry I rambled on.

Ted