Nope, no downside to removing the baffle other than the noise increase.

In theory, the drop in backpressure might reduce the pressure to the tank, but that's going to be minor, and shouldn't affect the engine running at all. Removing the baffle from OS mufflers is a common mod. Many engine makes don't have baffles in the muffler at all.