Hi WaveZ
The boat in your picture is a traditional scale-type runabout, not an out-and-out racing boat. What is it you want to do with the boat? - update to more modern electronics but be able to run it at scale-like speeds, or turn it into some kind of racing machine and make it run as fast as you can?
If you look at the modern versions of this boat such as the ProBoat Volere or Runabout, these both use 540/ 550 sized brushed motors today, similar to what you have now.
http://www.proboatmodels.com/Product...ProdID=PRB3050
http://www.proboatmodels.com/Product...ProdID=PRB2600
It looks like you have a 540 sized (36mm diameter) brushed motor in it, so a basic 540 sized brushless motor would be fine and should fit OK. You probably don't want to over-power the boat, it wasn't designed for it - older model boats have fairly flat bottoms to help them plane even with low powered brushed motors, which is not good if you try to make them go too fast.
I suspect you also have adirect drive to a soliddriveshaft connected to a propellor which is completely underneath the hull, with a rudder behind? (can you post a pic?) This will limit the size of propellor you can fit.
Do you have a nicad or Nimh battery charger already, or do you need a charger as well as batteries? If the first, newer NiMh batteries are powerful enough and easy to use. If the second, you could consider lipo batteries but probably not necessary for this style of boat.
For scale-type running (plus a bit more speed) with 6-8 NiMh cells you could consider something like this:
Motor:
http://www.hobbyking.com/hobbyking/s...idProduct=6534
ESC:
http://www.hobbyking.com/hobbyking/s...idProduct=7377
Water coolingjacket:
http://www.offshoreelectrics.com/pro...?prod=gnd-4196
Alternativewater cooling pick-ups (depending on access to inside of hull):
http://www.offshoreelectrics.com/pro...od=dh-aqub9520
Radio system:
http://www.hobbyking.com/hobbyking/s...dProduct=15141
Servo:
http://www.hobbyking.com/hobbyking/s...idProduct=3743
This would give more performance and eliminate the 4AA battery pack. The servoneed to be changed because you have old style Futaba connectors which are not used any more, and a new servo is probably cheaper than an adapter cable.
-yellow-bird