Yes adjusting the atv will do nothing to affect servo speeds but can make it twitchy if ATV is short and the mechanics are poor

You always want to use the shortest horn on the servo and the longest on the control surface you can get away with and max the ATV when possible, this takes full advantage of both the servo's resolution and torque.

Adding to the problem, I think your new "Snappyness" might be due to installing a higher speed servo also, try adding some negative exponential in the TX to smooth it out around center. I recommend you start around -40% and adjust to your liking from there.

And if slop is an issue, loose the crappy clevises and invest in better linkages,, like good quality ball links

good luck