I think that trying to drill a dowel out of foam, the bit would walk off the dowl and into the foam leaving you a large mess to deal with.

I would try a piece sharpened brass tubing with the correct ID and use the dowel as a guide to cut the broken dowel out. This should leave you with a clean round hole to epoxy the new dowel in.

There's my .02

DEG