Your symptons are almost contradictory, so the problem is probably more subtle than mere c.g. or surface deflection; two things can cause a difference in sensitivity are something in the plane moving around, like the receiver battery, or a less-than-stiff control system that is kept from moving a lot when the speed is high due to airloads, but can "over-deflect" at low speeds.
Another is exponential gain on the surface. There's two ways for that to work, and easy to do it wrong.
It sounds like you fly well enough to disentangle the situation, so proceed with a plan, changing one thing at time to pinpoint the source or the cure.