Lots of people fly under 20F (around -7C) with no problems. A little snow does not seem to hurt either.
Some pics:
http://www.trerc.fi/galleria/Alasella_2006/index.html

If you mean below -20C, it gets hard to keep yourself and especially the fingers in working order.