I would have sworn it was a 3mm, but I just checked my .40 FSR and I can confirm that it is not. According to my bolt guage, it appears to be an SAE 6. As you have already found, it is not a 6-32. My guess would be either a 5-40 or 6-40. You can get either at [link=http://www.mcmaster.com/#socket-cap-screws/=81s4se]McMaster Carr[/link].

Just found a Slimline muffler for the FSR and it has 3.4mm bolts. Don't know where you can get that.