Yup this is exactly it. I don't think the issue is that our fruit is larger, but just that the soil is much more degraded. 90-95% of all plant species get the MAJORITY of their nutrients from mycorrhizal associations and some, like many orchids, are even obligate. Plant roots are generally primarily meant for communication and connection with fungi, not to extract nutrients from soil.

But when practicing monoculture, the #1 threat is pests and so we do everything we can to fight them. This includes using sterilized soils, an overreliance on pesticides, and the usage of artificial fertilizers. The artificial fertilizers are a particularly big problem when it comes to phosphorous because for most plants they rely on the depletion of locally available phosphorus before they start the complex chemical dance necessary to make the association with fungi

The really sad thing is that soil inoculated with fungi can hold 50-100x more water than sterilized soil. Without it water tends to drain and we use much more of it. Combine that with the fact that most of the phosphorus in artificial fertilizers is not actually accessible so we end up using more water which carries this leftover fertilizer which causes massive cyanobacterial/algal blooms that have already led to the extinction of many fish species.

It's a race to the bottom all to maintain the practice of monoculture. The research on mycorrhizae is relatively recent (mostly in the past 3 decades), but at this point the existing evidence is astoundingly clear. Plants with mycorrhizal associations better resist drought, frost, soil pathogens, produce more nutritious fruit, and can even produce more fruit mass (depending on the species, some will end up producing more leaves instead but this can be inverted with some manual labor)