undefined | Better HN

0 pointsPittleyDunkin1y ago0 comments

Ah, so this is just marketing.

0 comments

(disclaimer: i wrote that post)

It is not. We rely on more than User Agents because they are too often faked, so it is not just marketing. There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent.

PittleyDunkinOP1y ago

> There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent.

Great! What are these signals? That seems to be the meat of the post but it's conspicuously absent. How are we supposed to validate the post?

signatoremo1y ago

> how are we supposed to validate the post?

Imagine you were a vendor who were trying to trick the author into divulging his methods. Can a stranger on the Internet be trusted?

dgfitz1y ago

I imagine if that information is disclosed, you won’t be able to verify it, as it will be bypassed… because it was disclosed.

1 more reply

superkuh1y ago

So user-agent and whois to see if it's coming from a plausible netblock and accept: strings, http header stuff?

j / k navigate · click thread line to collapse