undefined | Better HN

0 pointswhat11mo ago0 comments

Any scraper is also a “user agent doing work for users”. Which ones should respect robots.tx?

0 comments

Does the user agent fit the definition of a web crawler? If so, then observe robots.txt. This one does not, see https://en.m.wikipedia.org/wiki/Web_crawler

j / k navigate · click thread line to collapse

0 pointswhat11mo ago0 comments

Any scraper is also a “user agent doing work for users”. Which ones should respect robots.tx?

Does the user agent fit the definition of a web crawler? If so, then observe robots.txt. This one does not, see https://en.m.wikipedia.org/wiki/Web_crawler

j / k navigate · click thread line to collapse