Couple things you can do. I wouldn't recommend any type of silicon (it permeates all too well) as it will prevent gluing in the future should you have an accident.

1. sand the tube with 400 grit sandpaper.
2. tape 150 grit sandpaper to a 1/4" dowel and then wrap it around a few more times leaving about 2" of dowel exposed (this makes a home made
honing tool). Chuck this up in a drill and run it inside the sleeve pulling it back and forth. Really easy and it doesn't take much.
3. Use some baby powder on the tube and the sleeve(graphite works also but it is messy)

It only takes a few thousandths, so none of is takes long.