Joey,

mrasmm is right. Another thing you may be experiencing is blade wash. It's caused by the turbulence of your main rotor blades. If you are not commfortable flying 3 to 5 feet above the ground, you'll have to get used it while learning to hover just above the ground. Blade wash can cause your heli to bob up and down and side to side.
When you first started out, sliding around on the training gear, you probably compensated for this by giving the tail a little adjustment or a little right stick. When in a hover you'll do the same. Be careful not to over compensate, you want just enough to hold your heli in one place. Even when 5 feet off the ground you'll find yourself making tiny adjustment to keep your heli where you want it.
Hope this helped.

Ted