I am using ball links on the servo side. I considered putting ball links on both sides but I wonder if there might be some problems with it trying to twist the control horn bc it is pushing off center.

Not very much nylon bewteen linkage hole and the vertical bolt. You did not mention the size plane and surface area you have them mounted on. Suspect on a small control area it might just try to rotate the nylon on the bolt. Nylock nuts next to nylon would stop any rotation.