You should certainly use the sanding drum for final finishing if possible (depending upon the size of the drum and hole).

Drilling a small hole that a small rotary rasp/file can enter might well be the first step of a three step process--use the rasp to enlarge the hole big enuff for the sanding drum as the second step.

always,
tony