The pivot location is considered in the design along with airfoil shape and intended flight envelope. The balance has to be capable of resisting flutter at low speeds as well as high subsonic and supersonic. It is necessary to start with the balance on the pivot and adjust as necessary to compensate for shock waves, airstream disruption from the wing/ fuselage and max intended deflections. If you moved the flight control of one of those jets sitting on the flight ramp (with no hydraulic preload) it would surprise you how little force is necessary to move them. An unbalanced surface only adds to the necessary force to operate it and “G” loads only exaggerate the condition.