The nut is not what keeps the tip tight, it only keeps the screw portion from backing out. Take the nut off completly, and unscrew the screw portion. Make sure the tip and body are seated tightly, then screw the screw potion back on making sure the exhaust is pointed in the correct position (make sure to get the screw tight as this is what keeps the tip from rotating). Then screw the nut back on and tighten. My guess is that when you adjusted the exhaust, the body and tip were not seated tightly with the screw prior to tightening the nut.

This is assuming you have the stock OS exhaust.