I can't work out how an open protocol implementation of this could work without crypto: ultimately if it's just fiat, a business entity needs to be the payment processor who aggregates microtransactions and pays them out to content owners, this is the role cloudflare is playing.

The problem is microtransactions are not feasible in fiat, and to remove the aggregator role like Cloudflare means a huge amount of microtransactions from each potential crawler to each content owner. That's just too much expensive work compared to the current position.

I agree though, I shed no tears for crawlers, but hopefully we're beyond the naivety of honour systems - again, the thing crypto was supposed to be solving.

Forcing big evil crawler entities to bend the knee by hiding behind big evil CDN entities feels silly though.