A good rule of thumb to follow is to start with the recommended prop from the manufacturer. If you're not satisfied then modify as follows.

If you want more thrust on up lines, shorter take-offs, and more drag at low throttle settings, and are willing to sacrifice top end speed, then get an increased diameter prop. For each inch diameter you increase it, decrease your pitch by 1 or 2. For example, if you're currently running an 11x6, then try a 12x5 or 12x4. This will keep the engine in the same torque band, and top end RPM should be about the same.

If you want to increase speed, but sacrifice upline thrust and have longer takeoff rolls, reduce diameter by 1 inch, and increase pitch by 1 or 2.

Different types of props have different performance characteristics for a given size as well. For example, wood props are very stiff, but tend to be fragile. APC props are stiff and strong. That stiffness allows the prop to retain its shape better at high torque, and will give improved performance. MAS props or nylon props are strong, but not as stiff, and will bend slightly under high loads. They are less likely to break on ground contact, but may sacrifice some performance. Many people like them because they lend a more scale appearance.

Perhaps if you were more specific on your engine, we could give you some specific prop recommendations.

Brad