Possible causes not in any order:

prop unbalanced or badly warped
prop hub not square, causing prop bolt to bend
bent prop screw, move up to grade 8 hex bolt [5-40]
motor mount too thin or too soft
internal parts of engine getting sloppy

Generally speaking, I don't bother balancing 1/2A props. Even badly warped rubber props run smooth at 20,000+.

You can do something that will contribute to all around smoothness if the engine is running in ultra high range [above 24,000 for a Cox], shave material off the inner skirt of the piston.