Yes at high speeds small variances will be amplified.

The combination of the misaligned elevator halves (I hadn't thought of that) and the drooped aileron could be your problem. The right elevator being up compared to left would try to roll the model right... then trim-corrected at cruise with the aileron drooped... youhave a high chance of stalling the left wing taking off early.

Sounds like a string of small problems all adding up to bite you. When corretcing the ailerons... just raise the low one. (don't want to create a bad stall characteristic by having the ailerons droop) Gain more speed for takeoff as suggested above. Retrim for level flight and check by the NSRCA chart. (again with my noted expectation that it WILL pull up from a dive on its own, and trimming that out will be a mistake.)