Nice project! It is coming along very nicely and lots of work done already.
If you test the motor disconnected from any linkage, and you actuate the potentiometer manually, does it react to your potentiometer manual input? Does it change direction at some point? From your description, it sounds like when you are in the middle it does stop moving? It seems odd that you have to move the pot quite a bit before you get a reaction of the motor. I think it should be very sensitive as to keep the servo in position. Think of a regular servo, if you try to force it it does not even move yet and it already senses it and torques in the opposite direction to keep it there.
Regarding that middle spot where nothing is happening, is it a lag issue, or really a position issue? I mean, if you move the pot extremely slowly, it does the same thing ?