Sounds like you don't have the connections on the rudder right on (at right angles to) the hinge line; one side is slightly ahead of the hinge line (the one you pull when all lines get tighter) and the other behind the hinge line (one line slackens when you pull the other). It is very important that the attachment points on the rudder horns be on or slightly aft of the hinge line. The converse is true at the servo end, have the attachpoints on or slightly ahead of the pivot point on the servo.