The engine symptoms you mention sound like the engine's not getting enough fuel. It's too lean for some reason. You need to check the entire fuel system, and not only to make sure it's free-flowing with no air leaks, but also that it's not getting air agitated into the fuel by the engine's vibration. The tank needs to be insulated from vibration. 8,000 RPM with a 12.5 x 8 prop may be about right. We'll see about 9,000 on an APC 12 x 8. Even though your prop has only 1/2" more diameter, the maker, blade airfoil, and blade profile can make quite a difference. We've seen as much as 1,000 RPM difference on same-size props by different makers.

We'd still check out the fuel system, first, though.