Yes, and no...

Naturally when something works harder it wears out faster, it doesn't actually burn out the servo the next day or anything. I have seen people using 6v packs and servos that have had been in planes for at least 500 flight and still no problems.