OK, you have flown that airplane with the weight you just removed and have a feel for it's elevator response with the CG where it was with that weight. And removing it only moved the CG 1/4". Expect to feel a bit more response from the elevator with the CG moved 1/4" back, but don't worry that it'll suddenly become lots more sensitive and dangerous. Because it won't. If the airplane was comfortable to fly before, it'll be perfectly safe, but maybe a bit quicker now with just that 1/4" change.

The low rate elevator throw being more or less the mfg's recommendations isn't really a big deal. And low rate is usually setup on most non-3D airplanes to be the setting you fly most of the gentler flight with. It's very often that the high rate is expected to drive the airplane into snaps etc where the wing has to stall into the maneuver, and low rate is what's used to just fly around the sky.

Lots of experienced flyers adjust their elevator throws so that the high rate will be just enough to stall the wing at however slowly you intend to do it. Then they check to make sure the low rate deflection won't snap the wing out of their faster, round maneuvers. That gives them a setting they can use on the more violent manuevers, like snap rolls, and another setting that won't surprise them when they're not doing violent stuff.

One nice thing about our hobby is that we can edge up on things. Moving the CG a bit like you're doing is a very safe way to do things. Good thing about the CG is that it doesn't go over the edge suddenly on you. When you start getting it back into the twitchy zone, you will have felt it getting twitchy without doubt if you're only moving it 1/4" at a time.