Are these fuel airplanes , gas or nitro or electric ? What is the level of vibration from the motors ? If vibration , could be spots worn on the servo pots ( as mentioned) as it sounds like the issue is more near the servo neutral point. Another issue if higher vibration levels , could be some intermittent connection created in wiring or switches by the vibration.

anything loose in the fuse that would cause the COfG to shift ?