undefined | Better HN

0 pointssimonw8mo ago0 comments

Sure, but I still think it's OK to look at Apple with a raised eyebrow when they say "and our previously secret training data crawler obeys robots.txt so you can always opt out!"

0 comments

wat100008mo ago

I've been online since before the web existed, and this is the first time I've ever seen this idea of some implicit obligation to give people advance notice before you deploy a crawler. Looks to me like people are making up new rules on the fly because they don't like Apple and/or LLMs.

simonwOP8mo ago

I stand by what I said.

Apple are saying you can opt out of their training data collection using robots.txt.

But... they collected their training data before they told people how to opt out.

I don't understand why me pointing that out as "eyebrow raising" is controversial here.

hn_go_brrrrr8mo ago

It's not controversial, it's just not how the ecosystem works. There has never been an expectation that someone make a notification about impending crawling.

It might be nice if there were categories that well-behaved bots could follow, as noted above, but even then the problem exists for bots doing new things that don't fall into existing categories.

1 more reply

j / k navigate · click thread line to collapse