You are probably out of trim but a helicopter will not sit there and hover by itself even if perfectly trimmed. It will always try to fall of on either side or fore and aft. This is what makes hovering difficult at first. You have to learn to stop this unwanted movement as soon as it starts. Only practice and patience will get you past this stage. This is not an overnight process.