The plunger should move up and down, via the diaphragm. and the small brass T thing. The plunger should completely plug the hole, with the help from the spring, and when the engine is not running, or at idle. If that is not the case, then definitely there is an issue here.

Is the plunger in good shape? You may test it by taking the regulator out of the engine, take the screw out, fill the cavity with some glow fuel while holding spring in place. The fuel should not leak out from that hole.