That's why I don't like using tac rags - especially if I'm going to paint. For bare wood I use a shop vac with a brush attachment. I have a brush dedicated to nothing but bare wood and another for general clean-up. That way I don't get oil on the wood or gouge it due to whatever I happen to pick up with the vac.

Before I vacuum I spend a fair bit of time blowing dust out using my air compressor.

I will use a tac rag when it's gotten some use on something other than my models. By the time it's been sitting around for a while the wax has dried somewhat and I feel better about using it because it won't transfer the wax to the model.