You can have a rough idea about the thrust angle, even if you don't have an incidence meter.

The thrust angle has to be way out of wack to induce the reaction that you describe.

I bet your plane is rather tail heavy.

No harm in adding nose weight and see how it reacts, while keeping the power low.