I think if anything, it causes the bar to be super duper high to fully launch, which is why I’m very pessimistic on Level 5 driving happening anytime in the near future. AGI is not solved, not even remotely, and if death is the consequence, Waymo will probably dissolve before launch.
If all the AI did was decide you're unfit to drive, that it take over for a few minutes to get to the next stop and pull over and call the cops on you, we might save more lives by that alone.
This definitely would have been a consideration at some point along the journey.
You're naive if you think it wasn't.
Of course a waymo car will kill somebody at some point. Everybody knows it's going to happen, and there's no way google/alphabet hasn't essentially priced in that risk. If they were afraid of killing people, they wouldn't have gotten into cars in the first place.