^Yes it would
The formula goes I = mr^2
I is monent of inertia, m is mass, r is distance from rotation center.
So, the added weight does matter, even if it so tiny.
Reducing that weight will make the plane more agile.

About earlier thoughts of Bruckmann's Pandora:
The first one seemed to have a full flying stab and the one used in last WC with smaller canalysators seemed to have servos installed in stab. Information is based on photos in his FB wall.

When using fork type pull-pull wire ends detaching the connection is similar effort as unplugging servo cable.
So, pull-pull elevator is actually quite simple to build and use. It just seems to be more work than it is.
When it comes to the top level the changes in elevator trim (when temperature changes) seem to be noticeable with push rods.