The limitation of prop size is related to the MOTOR / ESC combination, and the amperage they can handle.
Amperage is the Voltage divided by the Resistance (I = E/R). That means, if you have the same motor/esc combo that has 0.74 Ohms, at 7.4 Volts (a 2-cell) then they will run at
10 Amps. BUT if you connect that same combination to 11.1 Volts (a 3-cell) you will be running at (11.1 / 0.74 = )
15 Amps! That's going to let the magic blue smoke out of your 10-amp speed controller!
So, they ONLY way to allow your ESC to survive the increased Amperage is to decrease the prop size enough to reduce the load on the system. Oh, and stop guessing and buy a WattMeter. It will be the best investment you ever made.
http://tinyurl.com/6anu4j
Yeah, yeah, I know it's an oversimplification, but the results stay the same - if you're near the limits of your ESC or the battery's "C-rating", then you've got to reduce the prop size or pitch or both.