This may sound silly, but I've had this happen before. I had aileron servos do this on a 27% Extra model, and on elevators from another model. I've found that the connector that is disconnected (when removing the wing) must be getting dirty. I can engage and disengage the connector several times, and the problem goes away. Check to see if your connector and pins are dirty.

I'm not sure if there's a difference whether gold plated connectors are used, or not - I've seen the effect on both.