There could be several problems. Was the servo ever damaged in any manner? You could have a gear damaged (one or more teeth sheered off a plastic gear), an intermittent electrical connection internal to the servo or an intermittent open in the signal lead. From your description, it does seem that the problem is with the servo and not the other components. When the servo is acting up, does flexing the wires at the base of the servo do anything? Does a gentle shaking or rapping on the servo case make it settle down? It is not to difficult to open the servo up and look at the gear train.