undefined | Better HN

0 pointschc10y ago0 comments

By abandoning their AJAX crawling scheme as described in the OP, they are essentially saying that they will evaluate JS for all sites. Do you have some reason to doubt that?

0 comments

greglindahl10y ago

If crawling with JS costs 1,000X or 10,000X as much as crawling without, it's fair to say that even Google isn't going to crawl 100s of billions of pages executing JS.

As a former web-scale search engine CTO, my opinions are commonly surprising to folks who haven't built a web-scale crawler/search engine.

tracker110y ago

My own experiments/experience shows that recrawls happen about 1/3 as often and tend to lag a few days behind for JS content vs inlined/delivered content. It's helped a little by dynamically delivering the sitemap data, but even that only speeds things up a little.

My guess is they're putting about 1/10th the effort into keeping things freshly indexed for JS, but may well be devoting 2x the resources vs directly received content.

copsarebastards10y ago

> By abandoning their AJAX crawling scheme as described in the OP, they are essentially saying that they will evaluate JS for all sites.

No, they are not. If you even think that's possible you're fundamentally misunderstanding how search engines work.

j / k navigate · click thread line to collapse