Your motor was trying to draw more current (amperage) than the esc could handle. So it overheated. Batteries usually make heat through discharging too fast because your esc is drawing more current from them. The motor is drawing current at different speeds in different amounts. If you think about it this way it may help. Voltage is the amount of water going through a hose (the wires of all your components), Amperage is the water pressure, and resistance (ohms) is the size of the hose. If any of these are mismatched then one will over heat. Don't concern yourself with the battery for now. In rc cars the esc's have to be matched to the motor and the battery no more than the esc is rated for (cells). I hope my limited knowledge of electronics has helped...If not ignore all my effort...LOL