Believe it or not a lot of people don't crash when they are first learning to hover, most people have their first crash when they start learning forward flight. Once you start into ff it is so easy to loose your orientation if your not familiar with the basics. That's why a lot of people will tell you to make sure you are "very" comfortable with the basics, tail in, side on (both ways) and nose in hovering before you ever think about going into ff.
It's not a question of "if you will crash", it a matter of "when you will crash", everyone does it, it's just a matter of time. You sound like your doing fine, just take your time and you should be fine, but trust me, you will crash sooner or later, all you can hope for is it's later rather then sooner .

I learn to fly with an LMH so my first cash wasn't very costly. My first crash with a big heli was a Raptor 30, the reason I crashed (in my opinion) was because the heli didn't have enough power to pull out. I come down a little to fast when I was in ff and got to close to the ground before I started to pull out and the heli just didn't have the power I thought it had and it hit the ground. It wasn't a bad crash, it broke the blades, bent the spindle and flybar, but it was enough to break my confidence, it was a long time before I had enough balls to try ff again.