Depending on the location, the issue is the duration of exposure.

Yes, both activities carry risks (as does all R/C aircraft activity). However, think for a minute about the amount of time that the other pilots are exposed to the danger.

When someone is doing high-speed passes while I'm flying, I am exposed to the danger from his aircraft for maybe 1 second per pass, with each circuit taking maybe 20 seconds (?) - so I'm exposed to the danger maybe 5% of the time I'm out there. Conversely, when someone hovers their giant-scaler right in my face they tend to do it almost CONTINUOUSLY. Thus my exposure, and consequently the overall risk, is VERY much higher. Now, if they would move that hover to somewhere else (away from me) then my exposure in this case would actually be less than from a high-speed pass.

YMMV. For what it's worth, I'm not against either high speed passes or hovering. I do (or try to do) both myself (sometimes quite badly), but I do my best to ensure that each is done either when I'm up alone, or done sufficiently far out so as to reduce risk to other pilots on the flightline. One thing I've never understood about some of the guys doing the very-close-in hovering in my club, is why they refuse to do it in front of themselves, and instead insist on doing it right in front of one of the other pilots. There would probably be less complaints if they would do it in front of themselves...

Regards,
Gordon