You simply steer away from those that are easy to misidentify. There aren't that many species in any given area, and those that look like poisonous are few, and well known.
In my childhood, when we lived in real suburbia, with woods in 5 minutes of walk or right at the door, we routinely gathered mushrooms, and never had a problem.
We stopped it when we moved to other places, or probably because adolescence came around -- probably, parents had taken us for an interesting kind of walk, and then we'd not want it anymore.
Right now I can only recall some names of mushrooms and only sure of one species, so I won't risk at all.
