If you lift the flap sightly does the buzzing stop? Is this not just the weight of the flap putting a small load on the servo? As the flap lowers the load decreases. It will all be completely different once airborne. I really wouldn't worry about it. Unless it's that strained sort of buzzing if you know what I mean.

Alex