The reality is that because the ailerons are easily effective, the airframe won't impart much load on them.
Another way to say it is that it's easier to roll the plane than turn it.
The surface area is not the determining factor.