It's probably worth emphasizing that this is risk with home soldering, where you're likely to eat and solder in relatively close proximity, and without being completely rigorous about changing your clothes and vacuuming up every last spec of dust.
Lead free solder works fine, so there is really no reason to take even a small risk if you are soldering as a hobbyist.
(And yeah, it's probably a small risk. By all means use leaded solder if you think the slight additional convenience outweighs the small risk of significant lead exposure.)