I think many people feel hovering is dangerous because they can't imagine themselves having complete control of a hovering plane. Just like any other maneuver, the better I get at hovering the less dangerous it is.

Certainly less dangeous than high speed passes!