If there is a visible, detectable delay between the time you move the stick and the time the surfaces move, you've definitely got a problem.

However, if the control surface moves immediately when you move the stick, your simulator plane is set up to be more responsive than your "meatspace" plane