I would probably "knurl" the bore somewhat as you suggest. Just be careful that whatever material you raise cannot flake off when you install the bearing, and subsequently get into the bearing.

AV8TOR