Been through this once by a noise complaint, and a land usage issue where the field was ordered closed better than 60 days. (note the field wasn't totally innocent)
You need to first thing form a club committe to find an alternate flying site.(just in case)
Make sure your ducks are all in a row. get a flyer over to where you say it's almost impossible to buzz people and try and buzz your flyer. if he says he got buzzed then get that flying site pronto.
If he says the airplanes were too high then you need to draft a letter to the city council, park supervisor, and police chief. Explain in the letter the insurance issue form the AMA, the AMA safety code. Invite these folks out to your field for a show and tell, demonstrate that it's impossible to buzz people. If Its possible to change the traffic pattern at your field to help in avoiding the site where the complaints came from then change it and have a letter and a sign posted before they visit your field.
you need to also go to a couple of city council meetings and meet some council people. invent them to your field for some show and tell. where you stress the safety issues..
Good Luck
.