For my planes I seal them all reguardless.

I'd say that in general that if you can measure it or if it's bigger than 1/16" I would seal them.

Take the ailerons for example in a high speed pull up from level flight.
If one side is 1/16" and the other Aileron is 1/8" obviously one is going to let some air slip thru more than the other.
This air is your lift or pressure and the difference will cause a roll motion.

Elevator will have the same issue. Causing a roll to one side or unevenness.

Rudder is not as critical.

Sealing can be as simple as tape or even clear covering for planes with trim schemes.