Your safety, especially avoiding being hit, effectively rests on mostly your own shoulders. That's why drivers learn defensive driving which teaches them they can get railed even when they have the right of way.
"I only look up from my phone when I hear an engine, so they should be stopping for me!" and the fact that the driver shares some of the blame isn't much of a consolation once you're laying in a hospital bed.
I also think it's a terrible justification for noise pollution.
That said, I'd be totally down for a city center that was just silent scooters, and where accidents got your scooter license revoked for 6mos no questions asked. ;)