RE: engine cuttoff
The battery is the more likely cause, in my opinion. The voltage produced by the battery will always drop under load. If it drops below the low voltage cutoff (LVC) setting of the ESC, the ESC will cut power to the motor. It is designed to do this in order to protect the battery from being over-discharged.
There is no need to guess about this. Get a wattmeter and look at voltage and current while running the motor. You'll instantly know whether the battery voltage is sagging below the LVC OR whether the current is too high for the ESC.
- Jeff