Could it be that the nut is bottoming on the threads before it actually starts squeezing the prop. You mentioned that the friction between the engine and spinner backplate does not seem like it's enough so I am assuming that after you torqued the nut you were able to turn the spinner backplate.

What make is the prop?