To take this a little bit further, since several have already explained how to find the offending servo, let's talk about checking the control surface and linkage. I would recommend taking the following steps:
1) Disconnect the linkage from the servo arm. Does the servo still buzz? If not, it's not a servo issue.
2) Check the alignment between the servo arm and rod end. If they are uneven, you need to realign the rod end. Reconnect the rod to the arm and test.
3) Disconnect the other end of the rod from the control surface arm and check the alignment. Realign the rod as needed and retest
4) If the servo continues to buzz, disconnect the rod at the servo arm and manually move the control surface. If there is dragging, disconnect the rod from the control arm and move the surface. If it moves easily, you have a push rod issue. If the control surface takes effort to move, you have a hinge issue that needs to be corrected

Just remember, it could be one of the above or a combination that is causing your problem