I do something similar.... I drill the hole for the hinge to the correct size... and then grab the next larger or second larger drill bit and "counter-sink" the hole....
making the hole bigger close to the hinge so the hinge can seat in a bit deeper....


make sense?