Much of why bigger flies like it is more stable has to do with the moments of intertia. Small planes need lower wing loadings to fly well so they tend to be very light. And what weight they do have is carried in close to the true Center of Mass. Take a long broom handle that might weigh 1 lb. Then take a short 2x4 that also weighs 1 lb. Hold each at one end and swing them back and forth. You'll find that the short 1lb 2x4 is easier to speed up and easier to stop than the broom handle.
It's the same thing with big models. Even when built lightly such as is the case with 6ft and bigger old time free flight models adapted to RC they respond to control inputs with a slower and more stately motion than small flitty models around 3ft span.
All that other stuff mentioned also enters into it as well. But even with two wildly aerobatic models the smaller one will be more "nervous" than the bigger one due to this mass-moment issue.