I agree with Evan that you probably have GeeBee floats.
GeeBees seem to provoke love or hate. Nobody seems to have mild feelings about them. I am not an admirer. Even if they were plywood/Balsa floats: Hollow floats are a great place to store water.

You will be WAAAY ahead if you will install a set of styrofoam floats. It may be a little work, because they don't come finished. But foam floats are (1) much more efficient in getting the plane airborne (2) incapable of absorbing much water (3) pretty darned sturdy(4) Very lightweight. I have an 8-year old set of styrofoam floats on their 5th airplane. They look ratty, but they still work.

FoamFloats.com. RnZ models, Climate models, PlaneFun Floats are some of the vendors offering foam floats. "Seaplane" here on this forum is PlaneFun floats, and he makes brilliantly engineered floats at astonishingly low prices. I'll cut you a set of float cores and send you detailed instructions for about what he charge to do all the work.

Evan also rightly suggests that you check alignment. The floats could be crooked, or your water rudder may be out of adjustment. The WR should not touch the water while your plane is going fast, anyway. It should not stick down from the transom any more than the height of the step of the float. It's for taxiing, only. The rubber-band retractable version sounds as if it is either an ERNST or Williams Brothers rudder. Either is OK, Ernst is more scale-like.