I just use an X-acto knife and square out the entrance of the drilled hole. Test fit the hinge and trim more if required. It's quick and easy and I think it's kinda fun.

Grinding the round hole out bigger will work but I always go for as perfect a fit on everything I can because a tighter fit equals less glue and less glue equals a lighter airplane.

Dave