Yes, restrictions are a problem where they’re a problem. YIMBY can help.
They’re not the problem. You could not only ban all zoning/building restrictions but make it legal to shoot people who suggest or support them you’d barely begin to touch the problem.
Incentives regarding what to build (or when or if) are as big a piece of the puzzle. And the fact that the very simple and hugely microeconomic concept of marginal returns rarely (if ever) makes an appearance in this conversation is a serious indictment of the economic seriousness of its players when it comes to how supply gets built. Cost disease also matters in construction. There are lots of moving pieces.
Even when supplying isn’t socially restricted, there are market forces that contribute to rising costs.