You bring up valid points. But comparing a pilot's attempts at maintaining attitude/altitude to servos holding the control surface fixed is not necessarily a valid comparison. When test pilots test stick fixed stability, they hold the stick/yoke rigidly in place, to see how the airplane naturally responds to this condition. They are not applying corrections, as this would mask the stability of the airplane itself and make it impossible to isolate. You can't measure a phugoid if you are not allowing it to occur by damping it with out-of-phase inputs.

Conversely, when they test stick-free stability, it's hands-off, again to measure the airplane's natural stability response, unaided by the pilot.