I understand your apprehension, but the FAA's 5 mile rule isn't a rigid wall. It doesn't say no fly, but rather requires that you notify the airport operator of your model operations. The only thing that has changed is the perimeter, formerly within 3 mi radius of an airport, now 5 mi radius. Many model fields have been operating for a long while within a 3 mi perimeter in cooperation with the airport operator to avoid conflicts. If your field has been operational for some time at 4 mi distance without any history of interference with traffic at the airport, you just need to tell the operator that, and unless you are talking to a real hard-ass (improbable), you are unlikely to encounter a serious threat to loss of your field. I'm confident you can do it, but wish you well in the effort anyway.
BTW, are you a TUFF member? I am, but my TDY schedule hasn't put me in the area with time on my hands since I joined....