When does a shaved prop loose it's efficiency on speed?
When the thrust the prop can deliver at speed becomes less than the drag of the airframe, the return in speed will start to diminish.
I ain't a rocket scientist, so I can't give formulas and calculations.
Just use Dustflyer's analogy..you go so fast with a 7.4 X 8, turning X RPMs. Now cut the prop down to 2" diameter with 18" pitch...RPMs are still way up, but your plane ain't moving at all.
You have to have enough thrust to overcome airframe drag, and drag increases exponentially with speed. So the faster you go, the more thrust you need. Trouble is, to get that speed, you still need RPM and pitch. At higher speed you need a lot more thrust than the little prop can develop.
Which means you need a cleaner plane or more HP.