1. Have you done a range check?
2. Is the antenna routed O.K.?? I.E. , not crossed anywhere or doubled over and run parallel anywhere?
3. Is the RX closer than 1& 1/2" to the ESC??
4. You may be right about overloading the ESC. After running it up, does the ESC feel pretty hot? If so, you may be overloading it. I think you're heading in the right direction by changing props to look at lower amperage draw.
5. You did not lengthen the motor wires when building the model did you?

Not trying to sound condesending because you sound experienced. Just going over some mistakes I made over the years that resulted in similar problems.

Good luck guy.

Chris