dhruv_ahuja

I am curious, what are people and organisations building with web crawlers, especially since now all of them seem to have LLM support for further refining the output from crawled pages.

Open source projects like crawl4ai[1] have a crazy amount of stars (~59k). This got me curious.

https://github.com/unclecode/crawl4ai

It feels like with AI, a lot of engineering practices that were de-facto standard have somehow been phased out. Or atleast that's the emotion I can see on reddit or on few threads in HN.

That led me to this question, what are some practices you feel have been phased out over time, that should make a comeback?

One off the top of my head: startups or even large orgs in general not hiring junior engineers. Who will teach other engineers in coming years if we don't train juniors today?