Ok, here's a silly question : does it do it the same when turning in both directions? I've noticed a lot of guys tend to always turn one way when going fast so I had to ask.

One of my P-47s would roll into tight left turns and out of tight right turns. It turned out to be a rudder trim problem. You might also check your lateral balance.

If it's only at high speeds, how about your aileron hinge gaps? Are they sealed? If not, then that would be my #1 guess.