we get to use the same facts frequently here:
Sec. 91.119
Minimum safe altitudes: General.
Except when necessary for takeoff or landing, no person may operate an aircraft below the following altitudes:
(a) Anywhere. An altitude allowing, if a power unit fails, an emergency landing without undue hazard to persons or property on the surface.
(b) Over congested areas. Over any congested area of a city, town, or settlement, or over any open air assembly of persons, an altitude of 1,000 feet above the highest obstacle within a horizontal radius of 2,000 feet of the aircraft.
(c) Over other than congested areas. An altitude of 500 feet above the surface, except over open water or sparsely populated areas. In those cases, the aircraft may not be operated closer than 500 feet to any person, vessel, vehicle, or structure.
(d) Helicopters. Helicopters may be operated at less than the minimums prescribed in paragraph (b) or (c) of this section if the operation is conducted without hazard to persons or property on the surface. In addition, each person operating a helicopter shall comply with any routes or altitudes specifically prescribed for helicopters by the Administrator.
You really dont want to show him the double edged sword that lets folks TO & L over his property
cause he will build a structure on the property line
that will prevent you from doing anything on your property (other than TO & L) within 500' of his gazebo.
Loops, rolls, hovers are not TO nor Land, so you cant do that on your property within 500' of the structure on his property.
Or
we can say the reg doesnt apply to models... no regs protection nor restriction