Tail rotor drift doesn't help either. This happens when the heli is in a hover and forces are equal (lift=weight and thrust=drag) with the exception of tail rotor thrust. In the Axe's case T/R thrust pushes it to the left.

Turbulence can be another problem close to the ground. It causes all kinds of havoc as the air deflects off of irregular objects adjacent to the heli. A clear, level area will make hovering much easier.

IGE (In Ground Effect) hovering does have one redeeming quality because it requires less power for a heli to hover. This comes in handy on full size birds.
Hopefully this will explain some IGE/OGE differences. Ground effect increases lift when the heli is within about 1 rotor diameter from the ground. At that distance it's close enough to compress the air between the ground and main rotor. Unfortuntunely if you pitch the heli with the cyclic it'll shoot off in the direction it's pitched. Like the heli was on a compressed spring. The real benefit of OGE (Out of Ground Effect) hovering is pitch changes do not release that column of compressed air and as Heli Bat said this makes OGE hovering easier.

Heli-Bat you were explaining tip vortices. This happens at the rotor tips because the air above the rotor is at a lower pressure than below it. This also means no lift comes from this part of the blade. Tip vortices are at there worst in a hover, but they don't effect stability as much as lift . In fact tip vortices seem to have little effect on RC helis because most have an excellent thrust-to-weight ratio.

Hey Carfanatic as far as mechanical trim settings go the best advice I can give is make sure the swash is flat and level in the neutral setting (right stick centered) because my heli's wasn't. Factory test flight don't think so. I also had to increase tail rotor trim so it would stop spinning about it's vertical axis.

Hope she flys smoother for ya