I balance as many as I can, avoiding doing so only if I feel lazy.

Yes it helps on piston engines.

An out of balance prop may be erroneously installed on an engine adding to the vibrational effects of all of that moving mass.

Someone once recommended to me that I install the prop after balancing at 90 degrees to the line formed by the cylinderhead/piston rod at TDC or BDC, when they saw my glow engine plane vibrating.

Sure enough that helped smooth things out greatly as the prop was no longer contributing to the harmonics of the vibration.