There is no pot adjustment in modern servos. We had that back in the 1970s. Haven't seen it in decades myself.
I'm inclined to think that you have a bad servo. Can you open it up and look for a missing tooth on one of the gears? That would give precisely the behavior you're talking about. Damage to gears is not uncommon, especially on servos attached to the nosewheel.
Andy