The RC model engines will not be damaged without a muffler. But two stroke engines tend to like a little back pressure though. Four stroke engines tend to work better with little to no back pressure. Usually the scale people use a in cowl muffler design. The in cowl muffler could be a Pitts type of muffler or something else. In the past we used manifolds with outliet pipes we cut to length so we could direct the exhaust to be just outside the cowl. Four stroke engines could simply use a straight pipe exhaust.

Since the RC model engines use muffler pressure to help pressurize the fuel tank as a aid for drawing fuel, having some back pressure is a good thing then. Now if noise is not a factor at your flying field or area, then you can leave the muffler off. But for our modern engines since we use muffler pressure, having some kind of a exhaust that provides some back pressure for pressuring the fuel tank is a big plus. Although one might experience a slight power loss, using a tongue muffler like the control line people use is a good way to go. One of the exhaust manifolds like Tatone and Slimine sold works too. One could make one as well, I think Dubro still sells the blank exhaust extensions, you can cut it to length as needed. Fabricate a simple baffle restrictor for some back pressure and it is good to go.

Now if you are flying a gasoline engine that has a pumper carb, then muffler back pressure starts to become more of a moot point as you don't need to pressurize the fuel tank then. But the engines tend to like some back pressure though. But one could still run just a straight exhaust stack though. But the bigger gas engines can be quite loud, so you only want to fly them where noise is not a issue.