Under normal circumstances, you're right.

However, there is a circumstance when a great deal of force is placed in that direction.

A lot of emphasis is placed on the "gap". Some people try to get as close to Zero Gap as they can. This is often a mistake. With the control surface too close to the airframe, it can act as a lever; pulling the hinge out of it's slot, or putting undue stress on the flex point.

I say, don't worry so much about the Gap, you can always seal it. Allow your hinges room to flex and your planes will last a lot longer. (Dumb Thumbs not withstanding)