The hex adapter is nearly impossible to remove. You will need to buy an adapter to replace it were one end fits over the motor shaft and the other end attaches to the motor shaft.
Here is what I purchased:
http://www.offshoreelectrics.com/products.php?cat=68
1/8 to .130 adapter
Somewhere on another page is an adpater others have used.
If you really want to try and remove the original adapter and you have a small tourch, you could try to heat up the hex brass fitting and pull it off. In theory, the brass should heat faster then the metal motor shaft, expand and come off.