Install the servo.
The additional drag will probably have more adverse effects on performance than 1.5 ounces for a servo. If you would rather save weight with the gear, then remove the retracts, cover the holes, and install a nice CF fixed gear on the fuselage. You may gain a three ounce savings doing that unless the fuselage need major rework for fixed gear mounting.
Me, I'd leave it stock and have a ball.
Mark