undefined | Better HN

0 pointslolinder1y ago0 comments

Robots.txt is for crawlers. It's explicitly not meant to say one-off requests from user agents can't access the site, because that would break the open web.

0 comments

Spivak1y ago

Yep, there's really two parts to this.

* Some company's crawler they're planning to use for AI training data.

* User agents that make web requests on behalf of a person.

Blocking the second one because the user's preferred browser is ChatGPT isn't really in keeping with the hacker spirit. The client shouldn't matter, I would hope that the web is made to be consumed by more than just Chrome.

j / k navigate · click thread line to collapse

0 pointslolinder1y ago0 comments

Robots.txt is for crawlers. It's explicitly not meant to say one-off requests from user agents can't access the site, because that would break the open web.

0 comments

Spivak1y ago

Yep, there's really two parts to this.

* Some company's crawler they're planning to use for AI training data.

* User agents that make web requests on behalf of a person.

j / k navigate · click thread line to collapse