I'm not sure if there would be any problem with General Aviation planes landing at an RC field but Ultralights are considered as "recreational vehicles" by the FAA thus as long as the ultralight pilot has the permission of the owner or leasor of the land and the land is not in a heavily populated ground area, it is not a problem.
Same is true with GA planes, the only problem is if you choose an improper field (too short, obstructions, rough ground) and crash. You would be held at fault for choosing an improper field. Bush pilots do this all the time.