Induced drag goes down as speed goes up, but it will still be the predominant form of drag on the aircraft at speeds below about 200 mph. (That REALLY depends on the value of C_Dp for your particular craft). In other words, reducing it will get you the greatest gains in drag reduction.

Lightening the aircraft's weight will effectively reduce the C_L needed to keep it aloft. Your trim will be different though and now you have trim drag to combat (off-design flight). The best option in this case might be to reduce the incidence of the main wing.... that may be going too far with modification to your aircraft though. You need to target a speed and lower the wing incidence to match it. I would think that the lower induced drag would really win out over any bad trim drag that you may encounter.