I find it helps to lower the nose into the wind to hold position, so if the wind is strong you will be closer to level. It then becomes more stationary flight than conventional hover, but you can lower the nose to go forward or raise it to go back. When it's windy I can't even try to torque roll, or I screw it up. But as long as it's not really gusty, stronger winds can be lots of fun for me.
Rich