Lead is not considered a class 1 carcinogen, but if you are pregnant, lead is known to cause birth defects. Also to impede the growth / development of small children. Lead itself is a poison always wash your hands thoroughly after soldering before you eat.
Like calvino said it's actually the flux or the resin and the other chemicals in the solder that are stinky and bad. You should always try to keep good airflow when soldering, for extreme situations use a fumehood or something similar (generally not necessary).

"lived another 20 years or so. As for my Dad 80, he's smoked since 15 years old (given up now tho'), worked in a chemical factory 40 years till he retired, breathed in god knows what fumes, much of this was before 'health and safety' officers stepped in to clean the factory's environment up, and yet now he's in great health for his age. Fortunately he re married a couple of years ago, and he's a new man again."
It's all a roll of the dice, you can smoke your whole life and it'll only have a 10-15% chance of actually killing you (will affect you in other ways though). If you are smart you learn to do everything you can to avoid dangers, but at the same time don't stress out or get paranoid about it since that too is harmful.