I agree, does sounds like you've got a leak. First place I would check would be the little blue rubber coupling between the exhaust header and muffler. It shouldn't have any holes or tears in it. It should two tie-wraps, one to clampit to the header and one to clamp it to the muffler.

Also check to make sure the header flange screws are tighten down.