The responses so far are good but one factor hasn't been mentioned. A more stable aircraft will respond to gusts more than a less stable one. For instance with a more rearward cg the pitching moment due to a momentary change in airspeed (in response to a gust) will be less than with the cg more forward. Likewise, an airplane with high lateral stability (dihedral effect) will respond to a momentary upset with a more positive motion than one with a flat wing. Inertia effects are there but within the range of sizes of R/C models, they are not very large.

To sumarize, to be least effected by gusty wind conditions, a model should have:

1. Lower overall stability (more rearward cg and less dihedral)
2. Higher wing loading
3. Low drag (clean aerodynamic lines)

While size may have a small effect, it is not the major factor. I have a little "Perky" with a Norvel .061 that meets the above criteria and it handles winds that many larger models have trouble with.