Leaving the crystal in the TX will just drain the battery faster. I doubt that it'll damage the RF block, but you never know.
Once, I forgot to remove the frequency block from my Futaba 7UAP, and I was "flying" RealFlight this way for a month. Nothing bad happened!