The pot is worn at the center point and the servo is hunting for center. When connected to the rudder, the momentum of the rudder carries the rudder through the center point and this sets up the chatter around center. By changing the trim setting, you're moving the center point. The chatter will probably go away when the plane is in the air with come pressure on the rudder.

Brad