I think lots of times it might save more than that, I picked a low number because I expect even modest savings will convince people to do it, especially if the impact on convenience is small.
For example: An UberX ride today generally costs between $10 and $20 here in sunny San Francisco (outside of surge). Nearly 80% of that goes to the driver! The premise of driverless cars leading to massive rides-for-hire is that driverless cars could massively reduce that cost -- probably from $10 - $20ish to $5 - $10ish.
So just there -- is ride-sharing really going to save $3? Presumably not for a $5 ride. Not for a $6 ride either. A $7 ride? Maybe! But if so Uber is sure as hell not getting a lot for driving ride-sharing.
Okay, so let's say that a driverless car would ordinarily cost you $10 per ride, and you can reduce it to $7 by sharing. First of all, note that this reduces the value of the service for you -- even if you have no safety concerns, it takes longer. Then add in the non-safety awkwardness thing. It's pretty close quarters for a relatively long ride. What if the other person is just socially awkward?
And then imagine if you will that it's someone that you feel -- rightly or wrongly -- is genuinely threatening. And you're cooped up in a Toyota Camry for 15 minutes with this person and absolutely no one to run interference for you.
For $3?
Also: if getting a driverless Uber costs $10, you should at least contemplate owning your own driverless car. $10 at current business write-off rates is about 18 miles. It sounds like in this hypothetical Uber isn't a great deal. You could realize most or all of your cost-savings by owning your own driverless car, and as a bonus you won't have to share your car with a stranger, and as a further bonus you won't get gouged at high demand times.
Point taken on how many people that would apply to if miles were cheap.
Also people use public transport without too much hesitation.
Public transit has (typically) more than one stranger. Indeed, crowds. That is, somewhat ironically, less threatening than being trapped in close quarters with a single stranger.