It benefits the person owning the driverless car, who can now do something other than drive while still moving from A to B. You seem to be insisting on thinking collectively, but that's not how people decide or act. It only has to benefit the owner for it to sell.
The rich will be the last to let their human driven cars go, but they'll also obviously be the first to adopt them, once they're safe. In this case, the rich other than the ultra rich, who can already afford cars-they-don't-drive... giving further evidence that, yeah, people want this.
Your argument would, for instance, seem to explain why cars never took off... why, one car hardly benefits anyone. There aren't even any car suitable roads, after all, and think of all the horses the smelly, loud thing will spook! But... that's not how people buy things.
The collective benefits come later. The individual benefits come first.
