Songkick's robots.txt has a few special rules. (opens in new tab)

(songkick.com)

1 pointsnintax13y ago2 comments

2 comments

It's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a little too fast).

nintaxOP13y ago

Did you notice the bottom 3 rules? Maybe I should resubmit and change the title...

j / k navigate · click thread line to collapse

Songkick's robots.txt has a few special rules. (opens in new tab)

(songkick.com)

1 pointsnintax13y ago2 comments

2 comments

dibarra13y ago

It's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a little too fast).

nintaxOP13y ago

Did you notice the bottom 3 rules? Maybe I should resubmit and change the title...

j / k navigate · click thread line to collapse