Yes, a weak servo can allow the surface to flutter, but the majority of flutters are caused by sloppy connections. If your pushrod is weak or the connections aren't snug, the surface won't be held firmly. Once the surface starts to flutter, it generates the forces that tear the wing up, or break the horizontal stab off the airplane. Sometimes the flutter destroys the servo's gears, allowing even more surface freedom to flutter even more.

What basically happens is that the surface is moved by aerodynamic forces in one direction. When the surface winds up maxed in that direction, aerodynamic forces have built up in the opposite direction and the surface is driven back the other way. This cycles and builds up greater forces with each cycle. It might wreck the servo gearing and get more freedom of movement, and that gives it even more power and speed. And depending on the strength of the wing or stab, the force of the flutter can tear the supporting structure apart.

Whenever a surface is deflected, it generates forces that try to twist the supporting structure. The ailerons try to twist the wing when they deflect for example, even when there is no slop in the aileron connections. Add some slop and oscillation, and bad things happen.