You might have bad motor/ESC connectors.....there is the possibility that one or more of them have a cold solder connection.
if that is not the issue they might be touching each other, i have had that happen and it makes that weird chattering you described.
I would start by re-saudering all the connectors. Chances are thats yer problem.