RE: ESC and Motor not match?
No... A ESC rated for 7.2 volts can handle more than 7.2 v up to a limit. That is not the problem. All 6-cell batteries will charge up to 8 volts or so and ESCs are designed to run at those volts.
I too think that your pinion maybe too large for this new motor. Try to get a pinion gear that is 3 or 5 teeth smaller and see what happens. If the pinion is too large, the motor will pull more amps to generate torque, which will over heat the ESC and motor. What I am not sure about without looking at it myself is why only the ESC is overheating, and not the motor with your new setup.
Another thing is you may have a faulty ESC and might want to see if the warranty can cover it. If you can, call Dynamite and let them know of your problem and see what they can suggest about the shutdown issue.
Also, what size pinion and spur does your Exceed r/c come with? What size motor did it come with? It might help with the problem if we know that. Look up that info in your manual or Exceeds website.