Denser housing could solve a lot of housing issues. The problem is getting denser housing built, especially in not-so-dense neighborhoods. This could be for any number of reasons - the accusatory voice in my head likes to think it's mostly due to NIMBYs ("not in my backyard!" or folks that don't want to live in higher density neighborhoods) because of their warped perception of these spaces: higher crime, higher traffic, unsafe for children, etc. Some of these views may be true, but it's not a given in every high density neighborhood. Safe, walkable, dense neighborhoods exist in many places already.
Another issue can be policy, and specifically zoning, which a lot of NIMBYs fight very hard to control. I'm no expert on zoning, but the general consensus among architects and pro-housing people is that it's holding back a lot of potential homes from being constructed. Check this out for some opinions on zoning: [https://www.washingtonpost.com/business/energy/can-the-us-ho...]
In my previous city there's a popular type of "middle housing" (not high end single-family, and not small apartments) called a dingbat. They at one point in time were crucial for filling the gap in housing but have now been regulated away. Check this out for info on dingbats: [https://www.youtube.com/watch?v=vlWvcsGlHA4]
For your neighborhood in particular, zoning regulations have likely limited height to 5 floors. Maybe folks in the community lobbied for this regulation to keep density down, or maybe your towns infrastructure can't support a higher density of cars (and this brings up building code and parking requirements that we have in America) - there could be many combinations of reasons for this density limit, but you should look into it! A city's history of zoning policy can be very interesting, as they oftentimes stem from decades old regulations or segregation.
Your optimism is valid - optimism is incredibly important for solving problems like these, especially when so many solutions exist! Implementing those solutions is usually the toughest part.
