Yes, it could be the servo. Could have a dirty pot or other problem. If it's JUST on the rudder and a VERY brief glitch it might be worth ONE flight test. First glitch, LAND and replace the servo. Replace the servo BEFORE flying if you have one available as they are not that expensive compared to a plane and safety issues.

Does the glitch seem to be range sensitive at all?