Grass is fine, but gravel can be a bit of a problem. Try to stay out of it if you can. Gravel hitting your prop can be bad. The rest of the plane is no problem, just the prop. As for flutter, it CAN be very bad at times. I have seen planes develop flutter on an elevator and 2 seconds later it departed the plane. Try flying a pattern plane without and elevator some time and you will see what I mean. It may not destroy the plane in the air, but it is not going to be in very good shape after it comes down. Landings are HARD without an elevator. (I know, rotten pun) I have never seen it happen, but I have heard of planes loosing a wing in flight due to flutter. That is why you should do everything you can to stop it before it starts and sealing the gaps in the control surfaces is a good start. Also make sure the control system has very little or no slop in it.
Ed