> Figuring out what pages are good is no longer a feasible task for software.

Of course it is. It shouldn't be at all difficult to train a large natural language AI model on SEO spam pages. But nobody will do it unless they can figure out a business model that makes it pay.

Or a non-profit organization is formed to do search spidering for the benefit of the whole world, much like Wikipedia or PBS.

I don't doubt the narrative as you relay it, but I question how hard this can be for one of the largest corporations with one of the largest engineering pools in the world.