[B]My club is b!tching about me hovering, they asked if I would only do it at either end of the field because it makes the other pilots nervous.
I'm with sailor. I never want to do anything that makes others nervous, and I'd appreciate the same courtesy. Folks that don't mind making people nervous REALLY make me nervous. It sounds like your club proposed a reasonable solution. Why on earth do you think it's jealousy? If folks say they're nervous, it's probably just that they are nervous.
This whole safety issue has little to do with pilot skill. People are people, and they validly fear propellers turning near them. Fear is good. Connectors come lose, radios die, even perfect pilots lose control of airplanes. Asking for an extra safety buffer is very reasonable. This doesn't just apply to hovering. Folks shouldn't be doing low passes over the runway, or manuevers that head straight toward other pilots and then turn away. Think at each moment, what would happen if I lost control of the airplane at exactly that point?