What heli are you using? If its a micro, then i wouldnt be too concerned--micros tend to do that. Try and get it steady the best you can with the trims. Once you get it a couple feet off the ground, you will be out of the "ground effect" and should get a more stable flight. You will just have to compensate with the cyclic stick in order to keep it from going forward and left.