Your motor is probably trying to draw more current than your speed contoller is rated for; causing the speed controller to shut down the current. The easiese fix is to install a larger rated speed controller and see if it corects the problem. Alternatively, you can install a prop that will require less power, thus allowing the motor to operate within the rating of your current ESC.

I had a similar problem with one of my planes. It is the Rare Bear from ThunderTiger. I installed a TowerPro outrunner, using a 15amp ESC, and tried to fly it with the stock 3-blade prop. At 75% throttle the motor would cut out, but if I kept the throttle below 75%, the plane flew fine. I replaced the 3-blade prop with a 2-blade, keeping everything else the same, and now full throttle is no problem.