Tom if I had to guess you actually really want to do it but perhaps your having an issue with peer pressure at your flying field or something.
People will look to be upset, anoid, angry and insulted with just about anything you do these days. F-them! Honestly. Its historical fact and it touched your family as it did mine. I dont deny it, hide it or make excuses for it. If someone cant handle it its on them not you. Your representing something factual, a swastika on a model plane should be no more or less offensive as a red dot is to some people in China.
Im half Italian, my Grandfather was an Italian tank commander in N Africa under Rommel. I have his medals on display in a beautiful shadow box on the wall and one of them is a black cross with a swastika in the center. The real deal.
Personally I hate it when guys dont depict the swastika correctly or not at all on German models, its a PC cop out and an attemp to soften history. There are no angels in war and every country that wages war should have their symbols banned if you want to go down that road.
Just do it and enjoy your historically accurate German model!