The good news is that Rustler hasn't changed all that much, and you will be able to find and upgrade parts at will. I'm not quite sure on the transmission in your Rustler. I don't know if there is a ball diff or a gear diff in there. My Traxxas usage/knowledge is from ball diffed Eagle/Blue Eagle and the TCP buggy. Hopefully someone will chime in with info on your model at hand.
As for not wanting to dump too much $$$ into your RC, I would just keep using NiMh packs to power that brushed motor until it poops out on you. After that motor of yours stop running, you could upgrade the ESC to handle hotter motors (brushed or brushless) or continue with "old school" brushed motors...I would suggest using at least
motors with open end bell so you can replace the brushes once they wear out.
Also, if you plan on upgrading to more powerful motors, you should upgrade those connectors you have on you Rustler/batteries/ESC/motor/charger.