Are you flying a coax or CP? If you are flying a CP I would recommend that you move your servo rods in a couple of holes on the servo arms. Go to www.cpbladecprepair.com for info on this etc. Just be sure to make a note of any changes that you make so that you can "go back" if needed.
Four feet is a pretty tight space for any heli and you may be having problems with blade wash or "ground effects" due to the limited space. Helicopters tend to get "sucked" into walls and the like when you get too close. I've been learning how to hover a CP in a 10'x15' space and it hasn't been easy.
I have now learned to take it up to 1.5 to 2 ft. so as to get out of ground effects. Also, it seems that most helicopters tend to drift to the left and mine is no exception. I have reduced this problem with some minor setup adjustments and by lifting the heli off at a faster rate. I think that lifting off at too slow of a rate was causing me lots of control problems. Good luck and good flying.