It is as critical as you want it to be... some people cannot build straight, YES, it will affect the flight characteristics, but as long as it is not off terribly, it will fly ok. You should be able to counter act the offset with either a slight adjustment of one of the elevator halves or the trim. Got a picture? If you are a perfectionist, you could run some support wires to pull/twist the tail feathers slightly so that everything is aligned. This could just be from the wood being slightly damp and twisting, it might come back to true if you sat the structure in the sun an hour or 2.
dan