4-40 rods should be okay. I'd just make sure that my hole doesn't remove more than 1/2 the wood diameter. And although I've used CA, I've also used epoxy and it might be a better choice.

Oh, and notice too that the dowel should be grooved so that the rods sit down into it about 1/2 their diameter. Helps prevent side to side movement.
Dennis-