undefined | Better HN

0 pointssunaookami4d ago0 comments

What an absolutely insufferable explanation from ArchiveTeam. What else do you expect from an organization aggressively crawling websites and bringing them down to their knees because they couldn't care less?

0 comments

wlonkly4d ago

ArchiveTeam (which is not the Internet Archive) aggressively crawls websites because they care a lot, because the website in question is about to go away.

Heck, I'd say as caring goes, ArchiveTeam cares more than the owners of the website, because in the ideal shutdown, the owners provide the data instead of forcing people to scrape it if they want to retain it after the site shuts down.

sunaookamiOP3d ago

They also crawl aggressively when the site is not in danger. They crawled my MediaWiki because someone else input the site in their bot and it overloaded the PHP process. I know that archiving is important but please, not like this.

472828472d ago

“Their bot” is a software anyone can run.

rossng4d ago

I'm curious to hear about examples of where this has happened. Because ArchiveTeam also has an important role in rescuing cultural artefacts that have been taken into private hands and then negligently destroyed.

tredre34d ago

Having a laudable goal doesn't absolve them from bad behavior.

Dylan168074d ago

It's a good reason to not worry about hypothetical bad behavior and wait for evidence of real bad behavior.

pabs34d ago

ArchiveTeam definitely do not intend to kill websites with too fast crawling, but definitely have done that unintentionally and always will stop/slow the crawling when it happens.

Even the distributed crawling system has monitoring and controls to ensure it doesn't kill sites.

tech234a4d ago

That page was written by Jason Scott in 2011 and has barely been changed since then.

textfiles4d ago

Why mess with perfection?

j / k navigate · click thread line to collapse