Ah, but you won't see those delta wing models flying in a pylon race.

Of course if they are flown in big wide patterns so they don't load up they will still be fast. Over a long enough straight line I bet they are faster than a high aspect ratio model. As Dick says it's a case of picking our poison and then flying it in a manner that leads with the strong aspects and minimizes the weaker ones.