Please check your wings carefully. You may have a twist in one wing that's being opposed by the ailerons, but as the speed builds up, it overpowers them.

Also check the fuselage/fin for alignment. If the fuselage is off a bit, or the fin is off a bit, the same thing could happen...as the speed builds up, the larger areas of the fuse and fin overpower the ailerons. Adding rudder then adds its control authority to the ailerons to help you level the model.