Gas/nitro:
You prop an engine to achieve a specific engine rpm at wide open throttle. Engines tend to have a peak power at a certain rpm, depending on unload, typically you shoot for a couple thousand under peak on the ground. Fortunately there are plenty of trade offs between pitch and diameter for sport planes, I like having enough pitch that I can fly a decent knife edge which limits the diameter. Then I shoot for the largest diameter for best acceleration and climb without loosing the speed for knife edge.

Once the best prop for the engine airframe combination is found, I then start to look at the engine thrust line, general balance and rough trimming. When it is correct, (does not climb or dive with power changes, and will pull easily into a straight up vertical with just a slight amount of right rudder). Then I start looking at inside and outside loops. With the wings level it should take about an even amount of right rudder to do an inside loop ast it takes left rudder to do an outside loop. The size of the he loops should also be the same size for this test. Once this is done, try a few props that are very close to see the effects and if you find one that performs better you will have to redo almost everything again, though by a very slight amount. After this fly about 20 flights to explore for slight trim problems that might indicate other problems.