Your method of blowing it off is a good start, but not nearly enough.

Give it a good wiping with a tack cloth, or better yet, I use a cloth sprayed with Spray Adhesive. It lasts much longer than a tack cloth and you can re-spray it if need be.

If I get any areas where the wood gets "fuzzy" (Where the ends of the grain lift from the surface) I mix 3 parts water with 1 part wood glue and paint the area, let dry and sand with fine paper - comes out smooth as glass.