It's a no-fly zone for an even better reason than just interference... Safety.

There's no reason to be doing low passes near a building, especially one that's 500yards out. At that distance it is hard to judge your distance accurately, and you shouldn't be flying over their property. There's probably something about this issue in the AMA saftey rules, and your flying field's club rules. As a Canadian, I have never read the AMA rules. We have our own set of rules (MAAC).