Denser housing could solve a lot of housing issues. The problem is getting denser housing built, especially in not-so-dense neighborhoods. This could be for any number of reasons - the accusatory voice in my head likes to think it's mostly due to NIMBYs ("not in my backyard!" or folks that don't want to live in higher density neighborhoods) because of their warped perception of these spaces: higher crime, higher traffic, unsafe for children, etc. Some of these views may be true, but it's not a given in every high density neighborhood. Safe, walkable, dense neighborhoods exist in many places already.
Another issue can be policy, and specifically zoning, which a lot of NIMBYs fight very hard to control. I'm no expert on zoning, but the general consensus among architects and pro-housing people is that it's holding back a lot of potential homes from being constructed. Check this out for some opinions on zoning: [https://www.washingtonpost.com/business/energy/can-the-us-ho...]
In my previous city there's a popular type of "middle housing" (not high end single-family, and not small apartments) called a dingbat. They at one point in time were crucial for filling the gap in housing but have now been regulated away. Check this out for info on dingbats: [https://www.youtube.com/watch?v=vlWvcsGlHA4]
For your neighborhood in particular, zoning regulations have likely limited height to 5 floors. Maybe folks in the community lobbied for this regulation to keep density down, or maybe your towns infrastructure can't support a higher density of cars (and this brings up building code and parking requirements that we have in America) - there could be many combinations of reasons for this density limit, but you should look into it! A city's history of zoning policy can be very interesting, as they oftentimes stem from decades old regulations or segregation.
Your optimism is valid - optimism is incredibly important for solving problems like these, especially when so many solutions exist! Implementing those solutions is usually the toughest part.
San Francisco's housing element should fall out of compliance in early 2023, making it possible to build things without dealing with the local housing cartel.
https://twitter.com/emily_sawicki/status/1580360066300928002 Discussion at https://news.ycombinator.com/item?id=33186186
I've also heard arguments for an urban model that focuses on smaller, more community-centric cities instead of huge urban centers like New York or LA. I don't have any primary sources for this, but I think the idea is to keep density low-ish, and increase the distribution of these urban nodes evenly across a region [https://en.wikipedia.org/wiki/Urban_village]. To me this sounds similar to the mundane suburban towns I grew up around.
Urbanity/density is likely the easiest[2] solution, but I'm sure it's not the only one. There are likely many thoughtful solutions that don't rely on density - I may look around to see if I can find any.
[1] I guess this issue applies across the entire spectrum of housing, which is why there's a housing crisis
[2] Easy is relative - obviously, this is has proven to be a very difficult problem to solve.
There are some interesting articles on the design choice, but it's a bit sad that we've gelled on this design.
The thing that sucks about the current model is that everybody seems to be making retail spaces designed for chain tenants which small businesses don’t need and can’t afford.
If it wouldn't be for political opposition most cities, undoubtedly, would upzone (permit higher density) areas near downtown and other hubs, but even in "progressive" cities progress is slow because of opposition of the people who currently live there (commonly called NIMBY's)
