I've had good luck on sloppy press fits by cleaning the surfaces with alcohol after sanding them to give them a little tooth.

Then you have you choice of adhesives. CA will hold fairly well but you will still be able to disassemble it with some effort. For a more permanent bond you can use Hysol or JB Weld.

Good Luck!