I think the problem is with the type of brake you're using. It's either a fixed load or it increases too slowly as torque builds up from the lower revs. Working backwards as you're doing from high revs and gradually increasing the load is fine until you get close to the point of max torque. Then with a fixed load the engine simply runs out of available torque to overcome the load so it slows down. This reduces the torque even more and it ends up stalling. If you started from lower revs and used a load that would suit the torque at lower revs the engine would reach this point then continue to gain revs and torque and then stabilise on the downward slope of the torque curve.

So what you need for a brake is one that increases the load very quickly compared to any RPM rise. This is what a prop does. It's load increases by the cube power of rpm so fitting a very large prop holds the revs well below the max torque point of the curve and reaches a balance.

You must use full throttle even for low revs. Closing the throttle restricts the amount of air that can be taken in and effectively makes the engine think it's smaller than it really is. In other words, a .60 at half throttle runs about the same as a .30 at full throttle on the same prop.