> I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known.

That’s not going to happen before Cloudflare is dethroned. See this recent thread for some perspective: https://news.ycombinator.com/item?id=27153603

And even if there’s no Cloudflare, large sites that people want to search will always find ways to block bad bots.

The only thing I can think of that might work is using crowd-sourced data, with all the problems that come with crowdsourcing.