In order for your setup to work properly, the cables must form a rectangle or as close to it as possible. The width of the horn at the rudder should be equal to the width at the bellcrank (if you are using one) or the servo horn. Some people prefer to keep the rudder hinge line inline with the horn others prefer to offset the control horn slightly behind the hinge line (positive ackermann). I tend the use the latter method and have had good results. When the surface is at neutral both lines are taught, as the surface is moved one cable is tight while the other goes slack. Since the servo is pulling in one direction and the wind is pushing against it in the other, the surface is rigid.
Start with the rudder at neutral, tape it in place so it doesn't move, and the servo at neutral. Adjust your cables so they are both taut. Ensure that the cables do not bind and that they are equidistant apart. This should solve your problem