it's just held in there by the compression of the plastic head, you probably just need to pull a bit harder or something. You might try the paper clip mod in the future =)

Also make sure to sand down the old main shaft to get it out through the bearing