undefined | Better HN

0 pointszenexer5y ago0 comments

This person used a new IP address for every single request, so that won’t work. And that’s a growing trend.

0 comments

Yep I can come at someone with datacenter/residential/mobile/etc. IP addresses, all incredibly configurable to slip around network blocks. Luminati and Proxy Bonanza are the services I've used with the most success.

Getting source proxy lists of high-reputation networks is just $$ and a simple API integration game anymore.

zenexerOP5y ago

There’s also the issue of CGNAT. If you rate limit too strictly based on IP address, you harm users who are stuck with CGNAT, especially in Asia and Africa. India is particularly problematic.

As for stuff like Luminati, if you’re being sufficiently sneaky, chances are you’re not going to snowball in the first place. I’m not sure why anyone would bother paying for Luminati to crawl sites like the one for which I work, but I have seen people use it to scam.

We can’t really be bothered to waste resources blocking well-behaved crawlers. Just keep it at a reasonable pace, respect errors (especially 429, but also 410 and 503), and ensure we have a way to contact you if things go wrong.

folkhack5y ago

Yep - I have the HTTP error code detection dialed into an extreme because it's dumb to run a broken scrape anyway.

Frankly just any errors - if I see more than say 5-10 jobs fail within a 2-3 minute time period things are designed to wait X time, try again... and stop if they're still encountering errors and ping me to come in and investigate.

Faulty retry logic is just as dangerous as the forked/distributed run-off situation.

CodesInChaos5y ago

I'd assume they use 1000 distinct IPs because the system scaled to 1000 instances and presumably many of them came from related IPs. So it makes IP banning considerably harder, but not impossible.

zenexerOP5y ago

More importantly, any reasonable IP-based approach would have a lot of false positives. What if there’s an RSS service like Feedly running on the same cloud?

CodesInChaos5y ago

Depending on your product, blacklisting all AWS IPs might be acceptable. For example my company has a VPN exit on AWS, which appears to be blacklisted by twitter.

j / k navigate · click thread line to collapse

0 comments

folkhack5y ago

Getting source proxy lists of high-reputation networks is just $$ and a simple API integration game anymore.

zenexerOP5y ago

There’s also the issue of CGNAT. If you rate limit too strictly based on IP address, you harm users who are stuck with CGNAT, especially in Asia and Africa. India is particularly problematic.

folkhack5y ago

Yep - I have the HTTP error code detection dialed into an extreme because it's dumb to run a broken scrape anyway.

Faulty retry logic is just as dangerous as the forked/distributed run-off situation.

CodesInChaos5y ago

I'd assume they use 1000 distinct IPs because the system scaled to 1000 instances and presumably many of them came from related IPs. So it makes IP banning considerably harder, but not impossible.

zenexerOP5y ago

More importantly, any reasonable IP-based approach would have a lot of false positives. What if there’s an RSS service like Feedly running on the same cloud?

CodesInChaos5y ago

Depending on your product, blacklisting all AWS IPs might be acceptable. For example my company has a VPN exit on AWS, which appears to be blacklisted by twitter.

j / k navigate · click thread line to collapse