Prop efficiency is THP/BHP thats it. Another formula is (Thrust * Axial speed) / (Resistance torque * RPM). The factors you mention do affect the BHP but since we are talking about two props being turned by the same engine at the same speed then the BHP is the same. You are saying the highere velocity brings the efficiency down, but it doesn't it brings it up. You are confusing the tip loss with high RPM. This is not a factor here as both propellers are turning at the same speed, in fact the tip velocity of the smaller prop will be less so there will be less loss there also.

Edit xx Getting confused here. If a prop of the same diameter is used and turned at a constant speed then the thrust goes up as well as the velocity of the prop blast with increased pitch. Buy if the prop is made smaller to keep both speed and power equal then the thrust goes down and the speed goes up even more than in the first case.

Using the Thrust HP calculator if one is to turn a 12-6 at 10,000 RPM then the Speed is 56.83 MPH, HP is .889, and thrust is 6.22 Lb. Turning a 11-8.5 at 10,000 RPM results in; Speed = 80.49 MPH, HP is .889, and thrust is 4.39 Lb. Thrust x HP is almost the same with the smaller prop being slightly less, thus vey slightly less efficiency when static. However at an aircraft speed of say30 MPH then the speed (leaving velocity - incoming velocity) of the larger low pitch will only be about 27 MPH (likely better because the angle of attack would improve) and with the smaller pitched prop the velocity differance is 50 MPH. Thrust should come down about as much as the velocity; so at speed, the smaller higher pitched prop should be more efficient. I am sure there are times when the opposite is true, but most of the time I think the smaller higher pitched prop will be more efficient except when static.