You're absolutely right about the reaction being wrong, but I guess that's just my inexperience. I only started flying two or three months ago. As you state, I should have flicked in to throttle hold mode but I've never done anything like that so my reaction was just to get the heli down without hitting anyone.
Hopefully if/when something like this happens in the future I'll be better prepared, but for now I'll just write this one off as more experience.