That is a LOT of weight! Are you sure its that much? Most of the stick on weights are in 1/4 oz increments. Anyway, yes you are ruining performance by adding that much. Move your battery as far forward as you can along with receiver. Do you have elevator servos in the tail? If so, you might look at moving the location of those (a real PITA). If all this fails to get you around 28% MAC, then I would look at using a larger engine. If you have to add weight, it might as well be power. By the way, the 32" is for one wing right?
Good luck.