It's definately some sort of interference, and it sounds like it's the old coupler problem. I was looking around, and there is a universal joint that'll fit the shaft to the coupler and I think that will work. The only thing I am unsure of is if the length matches up on it.
You can find the U-Joint on t his page, about 60% the way down. It's called "1/8" to 4mm (500/600 size motors to attach to a 4mm shaft", part #Bj17: http://www.offshoreelectrics.com/accessories.htm

You should email Steven V (you can find it on the offshoreelectrics.com contacts page) and ask him how long the u-joint is, and compare that length to what you need. If it's a match then you're a winner.