Question. Specifically which servos are you using? I had a set that got terrible at centering. They mostly did something like what your describing.They never did stay trimmed right from brand new and progressively got worse.They didnt return to the same position from each side. They were fast and strong but also cheap. The M955 or MG955 I think Tower Pro or one of the cheap brands. Ive seen the SG90 and MG90 do the same thing.
Look on the Monitor of your radio.If it is returning to zero from both directions this is what the Rx is telling the servos.If they arent centering its a pot or some issue in the servo.