Just push or pull them out using the larger allen wrench.
A wooden dowel would probably be better and less likely to damage the bearing.
Just be careful.