You are more productive in private offices with shared collaboration spaces rather than open office floor spaces.
High rents convinced the world this some how was a good idea.
Maybe, but I think the causal chain is complex. I mean, even at $5/sqft/month (which is really pretty nice office space) that's a tiny percentage of what one of the top-tier bay-area companies is paying an engineer.
I think open plan offices are in style; they look like what a scrappy startup might have looked like after the 2008 crash when warehouse space was super cheap (I myself rented some $1/sqft warehouse space in the early teens for my office. It was pretty great, but then it was as much workshop as office, and we had an under-mezzanine that was closed off and quiet for non-workshop work. Most of the space didn't have a mezzanine, and I certainly would have expanded the one that was there if I was staying a while.)
Of course, once you get people with money in you see these converted warehouses in SF with ridiculously high ceilings and no mezzanine; just a giant sun room or something.
I think what needs to happen is that the next scrappy startup that gets big needs to... start with individual offices. Or if they can't afford that, at least cubicles. I know people don't like how cube farms look? but as far as comfort and noise reduction, they are a thousand times better than an open plan office.
When I started in the late '90s, it seems that it was mostly a status thing; managers and the most important individual contributors got offices, while all but the most abject of workers got cubes.
This seemed to descend into a system where the higher status employees started doubling or quadrupling up in the offices, and the cubes on the floor slowly atrophied to the little bits of fabric you clip to your sit/stand desk these days.
Tldr - famous architect plus famous ad agency started trend
Any time anyone spoke everyone else could hear the conversation, so discussions tended to be formal and short. No "how are the kids" or other pleasantries.
Meetings would drag on, if only for the added bit of privacy you had in conference rooms.
There were some benefits, but overall I'm not a fan of them.