ha ha....That's a really funny name for a thread!
The anwer to the Q should I remove my pants? is always yes........uh, sorry.......
Anyway, I make my wheel pants removable by not just drilling a hole but making a slot (article in M.A.N. last year) and then I keep them off for practice and put them on when I want to look "spiffy."