Unfortunately, there isn't any (easy) mathematical way to come up with a starting point. Most of the existing combinations were determined using trial-and-error (many motors bravely sacrificed their lives), or by measuring the current with an ammeter.

What you don't want to do is exceed the physical limitations of the motor. As you put larger and larger propellers on a motor, you draw more and more Amps. Eventually you cross a threshold where the motor can no longer handle the current, and begins to get really HOT! When the motor gets hot enough, the windings start to melt, and the motor is ruined.

Use the Wizard in Motocalc, if you have a plane in mind. Simply limit the motor choice to the Speed 300, and Motocalc will do what it can to find a solution.