Although tangential to the billing issue, this is reckless. If you’re building a crawler of any kind, please, please, please prioritize ensuring this doesn’t happen so I don’t have to wake up at 3 AM.
I run the infrastructure for a moderate-sized site with probably about a hundred million pages or so. We can handle the HN hug-of-death just fine. But poorly-made crawlers that recurse like this? They’re increasingly problematic.
If your solution to fixing your crawler is “throw more concurrency at it and ignore the recursion,” and suddenly your requests start timing out, that’s a pretty damn strong hint that you’re ruining someone’s day.
From my perspective, this will look like an attack. I’ll see thousands of IP addresses repeatedly requesting the same pages, usually with generic user agent headers. Which ones are actual attacks, and which are just poorly-made crawlers? Well, if you’ve got a generic user agent string that doesn’t link to a contact page, and you’re circumventing rate limiting by changing your IP address, and you had the bright idea to let your test code run overnight, I’m going to treat it as an attack. At 3 AM, I’m not inclined to differentiate between negligence and malice.
This is happening more and more often, and I partially blame it on the ease of “accidentally” obtaining a ridiculous quantity of cloud resources. People deploy shoddy test code and go to bed. They turn it off in the morning when they see the bill.
It’s become so prevalent that our company has come up with an internal term for these crawlers that spin up a new thread/container for every page: snowballing crawlers.
Save a sysadmin: don’t snowball.
Oh, and include a useful user agent header so we can contact you instead of your cloud provider.
Puppeteer etc. are nice and all but if you can get away with raw HTTP requests grabbing and parsing the HTML without pulling down stylesheets, JS, etc. do it. It is WAY more efficient than requesting the full overhead for the user experience from these folks and threading out 5-10 workers to gracefully crawl a site this way doesn't typically cause things to melt down on your target's end.
You may be saying "well I need a browser-stack or evaluated JS to do my work" and you may be right... but honestly though 90% of this stuff is reverse-engineer-able with Charles Proxy and some basic webdev experience. Heck - I've even sandboxed JS from a target's site to generate tokens/etc to cut down on repeat requests. Even CAPTCHA stuff can easily be done without having to pull down full UIX overhead these days.
---
"Save a sysadmin: don’t snowball."
Implement thread limits, rate limiting, throttling, intelligent caching, and try to fit within your target's hosting capabilities without being disrespectful. Often I will "smear" large jobs over weeks worth of time so that it's only a trickle of traffic here and there (and to also fly under the radar... sorry).
Also - on the custom UAS: Unless you're trying to make it easy to get blocked/identified then don't take this advice. Let's face it - this is a gray area for most. The best way is to not "snowball" and to make your scrapers indistinguishable from a reasonable stream of real users from real networks. I would never expect a sysadmin to contact me because frankly they aren't paid to.
---
One last thought - the people who are out there writing these bots/crawlers/etc. are often the lowest common denominator. They're the type that will get something "working" and hurry onto the next job because the nature of the work tends to be a ton of low-paid contract stuff. Also, at almost every place I've worked at in ecommerce that has scraping involved it's the bottom-rung dev talent that's assigned to the work.
Sucks, but near-100% I attribute your "snowball" situation to that.
I can’t speak for other sites, but we’re pretty good at picking up on crawlers that don’t have a unique UA. The problem is that we’re going to have a hard time differentiating your well-behaved crawler from more malicious crawlers, and you’re going to get caught in the crossfire.
> if you can get away with raw HTTP requests grabbing and parsing the HTML without pulling down stylesheets, JS, etc. do it.
If you combine that with the lack of an identifying UA, there’s unfortunately a good chance you’ll get caught in the crossfire during an actual attack. That being said, it’s good advice otherwise. If you’re trying not to be identified as a crawler, it’s really going to stand out, though.
> I would never expect a sysadmin to contact me because frankly they aren't paid to.
I am. Furthermore, as long as you’re being transparent about your activity (see: UA), I don’t mind working with you instead of your provider. I understand that writing good crawlers is a learning experience; mistakes do happen. When I send abuse reports, usually people just get a slap on the wrist, but not everyone is that lucky.
But, if your UA has contact info, I can:
1. Easily rate limit or block you until the issue is resolved
2. Contact you directly, explaining exactly what’s wrong
3. Easily unblock you once it’s fixed
Sure, I’m not going to be happy about it, but I’m going to be a lot happier than if you try to blend in—a situation in which I’m not going to have any sympathy.
Unfortunately, most sites don’t respond that way and would rather just block anything remotely suspicious. But since you can always change your IP address, maybe try with an identifiable UA first—please? :)
Edit: Also, a few recommendations to add:
1. Be prepared to handle obscure HTTP status codes. 503 indicates you need to back off. Frequent 500, 502, or 504 means the same thing. 429 and 420 mean you’re being rate limited; slow down. 410 means you should stop requesting the given URL. 400 or 405 means you probably have a bug. Any unrecognized 4XX or 5XX error should be flagged and examined so you can handle it better in the future.
2. You can send an X-Abuse-Info header and a generic UA if you want capable sysadmins to be able to identify you but want to avoid being blocked by inexperienced webmasters.
3. Don’t ignore abuse reports.
4. Try to be consistent and ramp up slowly. It’s harder to cope with unnaturally-abrupt increases in traffic.
Typically, one doesn't care whether the same page has been visited before. What one does care about is avoiding storing duplicate data.
If it walks like an attack and it quacks like an attack...
I am lacking in sympathy for the perp here, as being careless like this has probably caused problems and possibly cost significant money for a lot of people.
However, this is also a compelling demonstration of why cloud services should be required to provide a hard price cap option for safety reasons. I've heard all the self-serving arguments they make about how turning things off surprisingly might be unwanted behaviour and so on. If that's the case, the admin won't set a cap. But there are exactly zero circumstances under which someone who intended to cap their usage at a level that would cost single digits of dollars or remain within a free plan intends or wants to run something that costs four orders of magnitude more than that, and IMNSHO such predatory pricing models should be illegal (assuming that the charges aren't already considered unenforceable by courts under such circumstances; I haven't checked).
We do have various ways to combat these issues; like any website of sufficient size, we have pretty complex methods of detecting problematic traffic and assessing the risk of any given request or session. However, no solution is perfect, and with the number of broken crawlers we see, some will inevitably cause problems.
To be clear, we can adjust our code and block them—that’s not an issue. The issue is that I have to wake up at 3 AM to do it, and even if it’s blocked, dealing with that traffic can be expensive. This guy got his $72k bill forgiven, but don’t expect the websites on the other end to be so lucky. (Yes, yes, ingress bandwidth is often free, but it’s never that simple. Scaling up? Bezos takes a cut. More database traffic? Pay the Bezos tax. Replication of enormous logs to other providers? Bezos hungry!)
Negligence is negligence. If you get in a car and drive recklessly without proper training, even if you didn’t intend to hurt anyone, you’re not going to get a lot of sympathy when you mow down a pedestrian. Likewise, I have little sympathy for people who face enormous bills for abusing powerful tools.
That’s not to say cloud providers don’t have billing problems. The delays are unacceptable, and the budgeting tools are often unintuitive or, as was likely the case here, outright inadequate. But in no universe was deploying code that spun up a container for every URL encountered a good idea.
Should such a mistake result in a $72k bill? Eh, probably not. I doubt this person will make the same mistake again, even with the bill forgiven. Or maybe they’ll just blame Google and attempt the same thing on AWS.
I noticed Cloudflare is doing the same but 1 level deeper with XDP drop: https://blog.cloudflare.com/how-to-drop-10-million-packets/
This isn't a mistake, the design is their business model. While we don't have a specific formal definition and name for it in the category of dark patterns, I'd like to name it "scumbag billing," where we got scumbagged.
It is — kid you not — recommended to terminate your individual services to avoid additional billing.
Technical limitation of a trillion dollar company? I say scum bag billing
Instead I use DigitalOcean where you have droplet limits that you can set, and the ability the pre-pay if you pay by PayPal, and never enter my bank card.
If anyone from DO (or another provider) is reading this, any chance of pre-payment from bank cards? After reading enough of these articles, this could really swing a cloud provider choice for a small company. (Pre-paid gift vouchers would be cool as well, give someone $10 to spend for Christmas).
It's too bad that DO only allows pre-payment via PayPal. (I recently lost access to my PayPal account due to them having a wrong phone number - and I can't login to correct it, nor have any contact for support.) I'd love to remove PayPal as a dependency, and pay DO directly from a bank - but not give them permanent permission to withdraw from it.
It's a clear dark pattern that GCP/AWS does not provide a simple way to prevent such unexpectedly huge bills.
I can attach a gift card to paypal and use it.
I tested the server for about 5 minutes and was charged a couple hundred dollars for "spinning up " the instance. Something the AWS sales guy assured me on the phone would not happen.
I still dont know why I didnt appeal I guess I know better than to try.
However, setting usage limits would be a solution for both companies and hobbyists. AWS could then calculate the maximum spending per month that is possible with the current settings. I bet they would never build such a calculator and the necessary usage limits because it makes it easier for customers to optimize costs.
But to be honest, it’s somewhat surprising that companies are willing to take on the risk of unbounded financial liability if someone makes a mistake.
Most AWS users would rather lose money than data and service for their customers, and bills are easier to negotiate than trying to recover your infrastructure.
The in-between approach is to create rate limits (either per sliding scale or total), which exists for some products but is probably too complicated to implement for everything.
Shutting off services can mean destroying the customer's data with no way for them to recover it. That could be from terminated ephemeral disks, or a terminated database, or cutting off a critical upload stream into their instances.
Its a lot easier to reduce/forgive a bill when a customer makes a mistake than to recover their lost data (or loss to their business).
I imagine they would have been more forceful if it was a larger bill.
Never used one for this purpose but since billing happens after the fact (and monthly), AWS won't be aware of the limit until after the monthly billing occurs. They’ll just tell you the card failed and you have a billing liability to take care of (and give you some time to fix it) while still letting you rack up additional debt with services running after the billing fails; they definitely won't cut you off when you've reached the level that would meet the limit on your card (and couldn't even in theory without realtime notification of other charges against the card, even if they were inclined to.)
I've been pretty happy with Cloudflare, but at some point I added my credit card (silly me) and now I live scared of a DDOS costing me a lot of money.
If you're talking about something pausing service, maybe call it a billing cap for clarity.
In other words, the only way to access the "free trial" is to give a blanket promise to pay unlimited amount of money if something goes wrong.
There is no way I would agree to that, so I just closed the browser tab and forgot about the whole thing. That is, until this debate reminded me of it.
For an online service, implementing the cap should be quite simple, so if it is not available, I am going to assume this is intentional.
How many times do they have to do that? Because if it is a high number, they would be operating at a loss.
What other kinds of businesses or services let you run up a bill of tens of thousands of dollars and then say, "Ok, you made a mistake, you can take it back"?
Any educated guesses on what this compute might actually cost Google?
I assume they're able to do this because the fixed costs have mostly been paid for already and the marginal cost of the electricity, system wear, and bandwidth are negligible, but I'm not sure.
I'm not using the service anymore,
