For top-end RPM, you may need a different prop. The one you're using is quite a load. Try a plain 15 x 8 propeller. Any idle below 2,500 is considered "good". Use an O.S. Type F glow plug for best results. Finally, you may need to run several tanks' worth of fuel before the engine will settle down. The engines with the backplate pump would generally need that much running after a long period of non-use because the regulator diaphragm would tend to stiffen, and the running would let the fuel loosen it back again.
Your O.S. Max engine with pump should be set up in a slightly different manner than a non-pumped engine. Most modelers tend to get them set with the mixture too rich.One sign your engine's not adjusted correctly would be surging at full throttle. This almost always means it's too rich. Also, the idle may be too rich no matter what you do to the idle mixture. If the idle is leaned a lot, but it's still too rich, the high-speed needle's too rich and needs to be properly set.To get a good setting, you need to use a tachometer. Once the engine's had enough running to accept being leaned to peak RPM, use the tachometer to lean the engine to its peak.At full throttle, lean the engine until the RPM stops increasing. Continue to lean the engine to the first sign of RPM drop or detonation. When you get to that point, richen to the point just before you had the RPM drop or detonation and leave the needle there. DO NOT lean to peak and then richen your usual 1/4-1/2 turn.With a good-quality fuel, the pump system will insure that the engine receives enough fuel. The regulator in the system will make sure that the mixture stays correct as you change the throttle positions. Fly the engine to see how the mixture may change as the engine unloads in the air. You may have to lean or richen slightly, depending upon how it acts. Once you've found the proper setting, you should note the change from the absolute maximum lean position. Now you have a repeatable setting for your engine.Newer O.S. engines with pumps have a broader range for the needle setting, but you can use the same method for all of them.