You never want to run an electric motor out of the water for more than a few seconds. With no load all the electrical power will be converted into heat within the motor. This could be the OP’s problem, especially if he were running 4S.
The suggestions above are also good, but it is impossible to diagnose the issue from thousands of miles away. It could be as simple as a bad ESC, I’ve seen even good quality ESCs burn up when plugged in for the first time. Rare, but it happens. Were the motor or ESC bought used?
.