A 12 knot crosswind would likely give fits to an ultralight pilot as well. But their ability to correct by "seat of the pants" feel from being right there would be more immediate. Likely because you're out of that physical contact loop and have to rely on seeing a crosswind caused displacement before you can react to it is what is causing your issue. A pilot on board would feel it long before a video or transponder needle would show enough effect to make you react. In this matter us RC pilots DO share in the same issue since we react to our models being displaced and then have to deal with it. But once we see the model is moving in an unwanted way there's already momentum and significant displacement involved so we need to damp that motion and correct the heading or pitch movement. Again, a pilot on board would feel such a displacement during it's first accelerative moments and react to them before any significant unwanted movement occurs.

So your own flying of full sized aircraft is actually leading you to expect something that isn't there when piloting a remote craft.

Perhaps rate gyros such as wel use in our RC helicopters and I'm sure are used in a fancier package in the UAV industry could be used to replace the "feel by buttocks" seat of the pant deal for such corrections.