(Subtle things, like huge firetrucks parked straight across the road.)
The AI isn't good enough. I'm starting to suspect that current ML learning rates can't be good enough in reasonable wall-clock timeframes due to how long it takes between relevant examples for them to learn from.
It's fine to lean on other sensory modalities (including LIDAR, radar, ultrasound, whatever else you fancy) until the AI gets good enough.
I predict a cusp to be reached in the next few years when safety advocates flip from trying to slow down self driving to trying to mandate it.