All the posts I kept reading on here and on
http://www.largescaleoffroad.co.uk said that you should use a Y connector. At the top of the Y, you connect the battery pack and the servo. The connector at the bottom of the Y plugs into the reciever. That way, the servo is getting a full 6v instead of 4.8.
I'm not sure what happened with mine as I can see no way to reverse polarity with these connectors.