Turns out my set screw was a little loose, I don't know how I didn't lose it like that, but I didn't.

Maybe that was the cause for the chewed up case. As you can see in the first pic, the set screw is riding right under the lip of the case. If there happened to be any dirt flying up in that area while driving, it easily could have helped in the grinding condition.

After I re-tightened the set screw, it wasn't riding as close to the case.

Could the whole problem be as simple as loose set screws?