You have to consider that the servo is square in the servo tray which would make it square with the movement of the pushrod at the center or zero point of the servo arm (note Overbored's sketch). That's why it's important to square the servo arm up at zero or the center point as my sketch showed. If it is not square, the arm starts off with plus or minus of the zero point so the arm will move more in one direction than the other, thus differential. So, with the servo zero'ed, and the servo arm square with the servo, the pushrod becomes square with the servo arm and it will move equally as much forward as rearward from center.

So, yes, it matters.

Note that on trainers, with one servo in the center, the servo arm on the ailerons are not square, but they are angled forward (uses the five part servo arms where the two in use are not square with each other but are at an angle to each other, more forward of center). This intentionally induces differential which helps prevent adverse yaw.

My my, we're getting technical.
CGr.