Use a small sanding drum in a Dremel tool. It grinds out quick and easy. Don“t use a drill bit as the chances of the drill bit "catching" or grabbing in the material and then breaking the adapter are high.
Also see here:
http://www.rcuniverse.com/forum/m_7239182/tm.htm (Posts numbers 20, 21, 22)
AV8TOR