There may be a couple of reasons for this.
First:
by preference, use not the transmitter for lowest idle trim, but the idle set screw on the carb, and switch off the engine using the ignition kill switch. This allows for a very reliable idle setup. For a faster idle, you can use the transmitter trim. This system is used in motorcycles and cars as well, and is very reliable.

Your problem can arise from either a slightly binding throttle pushrod, or a critical setup of the bellcrank angles around idle. Set up the carb belcrank to pushrod angle to a maximum of 45 degrees accute angle, with the servo bellcrank almost in line with the pushrod. This gives a very stable idle position. The remaining servo travel should just be able to kill the engine.
Setting the carb belcrank more accute, like 30 degrees, increases the effect of pushrod play or binding, causing the problems you describe. Even a slight temperature change or play has a large effect on the effective length of nyrod pushrods. Since the carb lever is the rod's free end, idle is bound to vary.