There is no magic "center" for servos, as I think Gus was saying. So holding the servo off center is no different from it holding at any other point.

However, as Minn said, your controls surfaces do have force on them when deflected, and even some force on them when "not deflected", In fact, I doubt you really ever "know" where the "zero force" point is for a control surface. The servo also has to work against the weight of the control surface and any forces created by the hinges or anything else. And if you add in things like turbulance that is constanly trying to "flutter" a control surface, I think it's safe to say that a servo in flight never has 0 load, no matter where the surface is. The good news is that the load is going to be small, and like Minn said, it's not noticeable.