RE: Nitro hammer..needs new engine
That engine should be just about a bolt-in. Use your existing parts as they will work fine. The only trouble that you might have is the engine lug holes and that is a big might, no big deal to take care of. The header could present a problem but I can't say for sure.
What you are probably seeing with your boat on flat water is chine-walk. Lower the prop on the shaft if you can. Going with the .18 you might find that the prop is going to be too small so figure in going to a larger one.
Converting over to a surface drive is a lot of work, you might want to save that for during your winter downtime but could be done in a couple of days otherwise.