Best bet is you're needing practice, lots of it, and that is not meant as an insult at all. Many have been messing with 3D for quite a while (years). It's not something you ever perfect. It's something you're always working toward. It's about building stick time - the more you commit, the better you fly.

I don't think there is a proper "set up" for hovering. I think proper setup depends on how the plane does everything else you expect of it - then you hover it.

Down elevator requirement when inverted might be interpreted as a little nose heavy.

Good practice conditions are when there's a slight breeze. You can kinda lean the plane into the breeze to help stabilize it. Could be on knife edge or inverted, but when just getting used to it, it might be easiest to have the plane headed into the wind and just play with it. Helps if the wind is coming from your left or right. Coming at or away from you a lot more difficult. Use power to control your altitude, and once you get it vertical, just tiny stick inputs will generally hold it in place.

Stick with it!