Seconding this. My team meets annually for several days, at a conference that gives us plenty of social time together in the evenings.
As you said it's hard to quantify the value, but anecdotally I notice it most in 3 (distinct but somewhat overlapping) areas:
(1) Overall morale - everyone enjoys work more when you have a good relationship with your coworkers, so people are willing to do more than the bare minimum. People approaching burnout feel more enthusiastic about work afterwards.
(2) Everyone is more inclined to help each other out with tasks outside of their routine but within their skillset, reducing bottlenecks.
(3) Similarly, you develop a better sense of each other's personalities and skillsets in a way that's much more difficult when remote, so communication is more efficient, and collaboration more effective due to that increase in understanding and empathy.
