And you definetly placed the balance point where it's called for?

And did you do a "top view" alignment of the wing to the fuselage and fin to the fuselage centerline? By this I mean you lay a centerline lightly along the top of the fuselage. Then you lay the wing on and check that it's equal span on each side of the center line to within 1/8 inch. Then you measure diagonally from the tips to the centerline back by the fin. These diagonals must be the same to within better than 1/8 inch. Within 1/32 inch is what I aim for when I'm doing mine. Then you want to ensure the fin is glued on so that it's exactly in line with the center line. Once you've set the wing on evenly and completley straight you should mark the position so it's easily repeatable or even key it someway. This is a lot more fussing than just putting the wings on and sort of looking at it and saying "close enough".

ALso just for giggles what size and how many rubber bands are you using to hold the wing on? A common beginner's mistake is to attach the wing with far too few rubber bands.

If all these things we're asking about seem to be set right then it's likely that you're not launching it straight ahead in a smooth javelin like push. If you try to toss it haphazardly like a baseball it'll be rotating in your grip strongly and that'll react instantly upon release. A glider of this sort doesn't need much of a throw. Just a firm but gentle push straight ahead.