Wheel pants do have aerodynamic qualities about them. I won't fly a plane without them...but each his own. I beleive they cut down more drag, and that IMO is more benificial then the slight weight penalty of having them. If your tearing them off guys.........ummm...errrr.......could it be something your doing?
True, they can be a bit of a pain, but once you have the mountings secure, if you don't bump them around...on the ground, or elsewhere...they should stay put for quite a while. Also...if your just going to leave them off...what's next....spinner.....canopy....a few tears in the covering.....the challenge is having everything intact...go ahead...I challenge you to try and keep them on....and if they come off....fix them..and if they come off again...fix again...do that enough...and a few things could happen....#1. You'll get very good at fixing/mounting them, #2. You will get very good at keeping them intact on the aircraft, #3....you'll never never touch one again My 2cents. Now..go fly!

Really though..they are there for aerodynamic reasons I think.