fuel foaming?
From your desrciption the problem could be fuel foaming. If you're riching the high speed needle a full turn or more to get the engine to settle down that would be my guess. Use a balancer to get the prop in perfect balance.
Also watch the fuel line and you may see it turning foamy looking.