In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing.
I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and slower-scaling (unless explicitly overidden/allowed, of course).
If I accidently push very non-performant code, I kind of expect my service to get less performant, quickly realize the problem, and fix it. I don't expect a service to seemingly-magically detect my poor code, increase my bill by a couple orders-of-magnitude, and only alert me hours (if not days) later.
I'm not sure why you think this is a trade-off. In general cloud services automate operations. Whether they are faster is unrelated. Many are not--services that use object storage for backing storage can be orders of magnitude slower than equivalent software using nVME SSD.
It's not a "tradeoff", it's a product feature.
When you put it like that, it sounds like an awfully good business to be in.
I think too, in most cases, people would rather run over than cut service.
Also how would such a system work? Let's say you sign up for some API and what, set your billing limit to 500 requests per day. Let's say you're now hitting fabulous numbers / signups - but suddenly you start hitting that 500. If that shuts off your signups or what have you, you're typically going to be worse off than if you just pay the overage bill.
I know it sucks, but the first time you pay your overage is probably your last.
Step 1: You give people the ability to put in soft limits - "Warn me when I hit 500",
Step 2: You also give the ability to put in hard limits "Pull the plug at 10k" (caveat to both these things - you guarantee this at an eventual consistency level, like "Well you hit 500 but by the time our stats updated you were at 600",
Step 3: You introduce rate limits - "We're expecting 500 in a month, warn us if we hit 50 in a day or 10 in an hour".
Step 4: You introduce predictive warnings "Our statistics show you'll hit your monthly limit on the 23rd of the month"
Step 5: You put in predictive limits to allow scaling - "The last 3 months we've seen the following use trend, warn us if we exceed double that trend, cut off if we see 50x that trend"
You might set some of these limits or none of these limits depending how predictable your use case is.
Sure, you get alerted, confirm it's reasonable, and then change your limits. You're also describing how many APIs actually work.
I'll say there is also a difference between going from 500 requests/day to 1000 requests/day, where you might say "this is probably legitimate and I want to run over", and from 500 requests/day to 25k requests/day.
One is mildly inconvenient, and the other is potentially bankrupting.
Or obviously if you don’t think this will be a problem, you have control to set it to uncapped.
I don’t understand what argument you have.
I'm saying that the cloud provider shouldn't try to make assumption either way, and I'm definitely not saying that it should try to manage indexes for you.
If you are typically using X ops/s, and begin using 50X ops/s, the default should not be "this customer probably wants to spend 50x their previous spend". It should maybe scale up some percentage of previous usage, but definitely not into a range that would be considered anomalous.
> The choice of infinity scale tools comes with infinity scale costs, and so there’s a responsibility that engineers using these tools need to understand what they’re accepting with that choice.
Sure, but I have never once seen one of these providers make clear that using them comes with the risk of being charged "infinity money".
Computers do what you tell them to do. If you are totally clueless and don't bother to take even a few minutes to try to understand a system you are using, the results are going to be poor. Thinking any system can overcome total user ignorance is the thing here that isn't sane.
What the person in this article did is like opening all your windows and setting the thermostat to 74 degrees. It will use massive amounts of energy and just keep trying to heat the house 24/7. If someone turns around after doing this and claims there is actually a problem with thermostats not being smart enough because what if someone doesn't know leaving the window open lets cold air in, well, they shouldn't be allowed to touch the thermostat anymore.
In theory I agree, but this website features something like "how I nearly bankrupted myself with an AWS bill" on the homepage every month or so. People are blissfully unaware about the extreme costs they're paying to the scaling cloud providers that they often don't even need in the first place.
While I don't think services should block extreme spend all together, a monthly/weekly/daily limit would go a long way to prevent these stories. Very few services that abstract away performance costs have a good way to limit expenses. I don't know if that's intentional or if these companies just don't care, but it's infuriating to me.
It's fine to expose the same tool to both someone who doesn't know the difference between indexes and foreign keys and someone who's been building cloud infra for many years, but as a company you should be prepared to respond to your customers' most likely mistakes. This specific case would probably be hard to detect automatically, but so many wasted CPU cycles, kilowatts and forgiven bills could be prevented if someone would just send an email saying "hey, you've been using more than 10x the normal capacity today, everything alright?"
> If you are totally clueless and don't bother to take even a few minutes to try to understand a system you are using, the results are going to be poor.
Having a hosted system which behaves different than the underlying technology it's modelled on is not immediately clear. The realm of "things you don't know that you don't know" expands drastically with managed services.
> Thinking any system can overcome total user ignorance is the thing here that isn't sane.
It's never been suggested that this is possible. There is a large range of options in between "solve all user error" and "don't hand everyone a loaded foot-gun".
You could then get various interesting stats in real time as well as some pretty useful alerting.
The difference is that application performance metrics are generally available in near-real time, whereas billing metrics are 1) very platform specific, and 2) generally not even close to real time.
It's hard to react quickly when your platform has effectively transformed near-real-time performance alerts to delayed/rolled-up billing alerts (which would also be much more difficult to use to pinpoint where the underlying issue is)
If you create a horrible internal combustion engine, your gas station should not bear the costs.
+1 on the budget control, but I don't think there are good arguments in favor of slower scaling.
The ability to scale on demand is sold (and bought) based on the expectation that services just meet the workload that's thrown at them without any impact on availability or performance. That's one of the main selling points of managed services, if not the primary selling point.
Arguing in favor of slower scaling implies arguing in favor of downtime. A service that's too slow to scale is a service that requires a human managing it. A managed service that is unable to meet demand fluctuations is a managed service that can't justify the premium that is charged for it.
A scaling change that would be considered anomalous, and introduces an order-of-magnitude change over historical usage could be scaled more slowly.
> Arguing in favor of slower scaling implies arguing in favor of downtime.
Sure, I guess that in a limited scope, that is what I am saying. I would much rather have a short-term "downtime that requires human intervention" problem, then a long term "Johnny deployed bad code and now the company is bankrupt" problem.
> The ability to scale on demand is sold (and bought) based on the expectation that services just meet the workload that's thrown at them without any impact on availability or performance. That's one of the main selling points of managed services, if not the primary selling point.
I tend to disagree with this. Managed services are often bought on the expectation that they do not require management, deep operational knowledge, and are reliable. There's also often the trade off of upfront costs (either human or capex costs).
Scalability of obviously part of the analysis, but "scability" and "the ability to scale from 1X -> 100X in a couple seconds" are not necessarily the same thing.
Oh, that would be actually quite useful for learning things if the bill would tell you that it got so high because you stupid dump-ass didn't use DB indices properly.
I'm every time shocked how many people using DBs don't know about indices! Those people should pay such a bill once. They would never ever again "forget" about DB indices I guess.
Of course I'm joking to some extend. But only to some extend…
In fact it's like a big bullet point near the top of the docs page.
"MySQL requires indexes on foreign keys and referenced keys so that foreign key checks can be fast and not require a table scan. In the referencing table, there must be an index where the foreign key columns are listed as the first columns in the same order. Such an index is created on the referencing table automatically if it does not exist. This index might be silently dropped later if you create another index that can be used to enforce the foreign key constraint. index_name, if given, is used as described previously."
I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page.
When you're programming, how many compiler errors do you see a day? (For me, easily dozens, likely hundreds.) Do you think each one indicates a serious gap in your knowledge?
Along these lines: imposter syndrome is a common problem in our industry. One way it can manifest is junior engineers can thinking they're bad programmers when they repeatedly see walls of compiler errors. I think it'd help a lot to show them a ballpark of how often senior engineers see the same thing. [1] I know that when I'm actively writing new code (especially in languages that deliberately produce errors at compile time rather than runtime), I see dozens and dozens of errors during a normal day. I don't think this is a sign I'm a bad programmer. I think it just means I'm moving fast and trusting the compiler to point out the problems it can find rather than wasting time and headspace on finding them myself. I pay more attention to potential errors that I know won't get caught automatically and particularly to ones that can have serious consequences.
I think the most important thing the author learned is that failing to add an index can cost this much money before you notice.
Ideally the author and/or the vendor will also brainstorm ways to make these errors obvious before the high bill. Load testing with realistic data is one way (though people talk about load testing a lot more than they actually do it). Another would be watching for abrupt changes in the operations the billing is based on.
[1] This is something I wish I'd done while at Google. They have the raw data for this with their cloud-based work trees (with FUSE) and cloud-based builds. I think the hardest part would be to classify when someone is actively developing new code, but it seems doable.
But he didn't see compiler errors, he caused monetary cost to his employer.
When I deploy something that unintentionally causes a large monetary bill to my employer, then yes I do believe that indicates a gap in knowledge so I don't in anyway believe I'm being uncharitable. Or and this would be worse, a lack of caring. (Which is not what I think happened here though)
I won't respond to your imposter syndrome bit I don't really think it's relevant to my point.
The problem is that in the old days, not knowing about indexes left you with an underperforming system or downtime. But in The Cloud™ it leaves you with an unreasonably huge bill and that somehow as an industry we're accepting this as normal.
They'll have to clean up the mess which causes real business consequences that, and I've personally seen this, will directly impact bottom line and have no quick or easy solution to wiggle out of.
Maybe it's acceptable for products like this because the balance between good engineering and company health probably aren't as cut and clear but stuff like this always makes me sad because it's such low hanging fruit, it doesn't require any real effort, just basic curiosity around your job.
No. Nobody finds that "normal", that's just untrue. It's even the _whole_ subject of this blogpost: the bill was not normal.
I don't disagree that some people are overrelying on cloud services, but that didn't become normality, it's still a beginner's mistake
The author had two misunderstandings:
1) An index isn't created automatically
2) You get billed for the number of rows scanned, not the number of rows returned
Even if I noticed #1, I probably wouldn't have guessed at #2 for the same reason as the author.
And in the end, if the cloud provider wants to charge you for rows "inspected", this can't be buried in small print. That's unacceptable!
The billing must come with up-front, red capital letters warning, and must come with alerts when your bill is unexpectedly little high (higher than expected, not just 10x or 100x higher). It must automatically shut down the process, requiring the customer confirm they want proceed, that you actually want to spend all that money. And it must be on the cloud provider to detect billing anomalies and fully own them in case it goes the wrong way. This is the cloud "bill of rights" we need.
Because it's a well written, humble account of learning from a mistake then using it as an opportunity to teach others to help them avoid the same mistake.
If anyone leads a team, I hope they might learn from this approach, rather than just bashing on people and implying they don't deserve any attention because they made a mistake a more experienced developer might have dodged.
I never worked with a database that doesn't have foreign keys and it's not unthinkable to forget when you do for the first time, that foreign keys were what created indexes for you automatically.
A little bit of planning could have prevented that though :/
It’s core to our mission that our product’s pricing is accessible and friendly to small teams. Part of being in beta was us wanting to figure out the best pricing based on usage patterns. That work is nearly done. As the post mentions we’ve credited back the amount.
With any new product there will be tradeoffs and rough edges but the positives, like easy migrations and database branches have definitely outweighed any difficulties.
Could you share a little bit about what your thought process was in general when picking a database technology?
You call out "easy migrations and database branches" outweighing other quirks, so some pros and cons weighing must've happened :)
Is it easy, for example, to test things in your dev environment with realistic amounts of data, and to get an understanding of how the queries will execute, etc? These seem somewhat basic, and would've probably caught this (also kinda basic, sorry :) problem early. (As in discovering "why is this query that should be a few ms with an index lookup taking so long?" early on)
Having the database constrained as much as possible makes maintenance so much easier. Many bugs don't escape into production as they're caught by the database constraints. Those that do get out do less damage to the data.
I know scale comes with trade offs but that seems extreme to me.
Is due to fundamental differences in postgres vs innodb?
/s
It's not a newfangled serverless pricing model, but it's something I can reason about as a multi-decade developer of database apps. I feel comfortable that our users--mostly devs--feel the same way. We work to help people optimize the compute and storage down to the lowest levels that meet their SLAs. The most important property of the model is that costs are capped.
One of the things that I hear a lot from users of products like BigQuery is that they get nailed on consumption costs that they can't relate in a meaningful way to application behavior. There's a lot of innovation around SaaS pricing for data services but I'm still not convinced that the more abstract models really help users. We ourselves get nailed by "weird shit" expenses like use cases that hammer Zookeeper in bad ways across availability zones. We eat them because we don't think users should need to understand internals to figure out costs. The best SaaS services abstract away operational details and have a simple billing model that doesn't break when something unexpected happens on your apps.
Would love to hear alternative view points. It's not an easy problem.
https://www.hetzner.com/dedicated-rootserver/ax161/configura...
Draw that nice red slide all the way to the right. No, it's not storage. Yeah, it's actually affordable. Yeah, that was a sexual sound you just made.
You do have to be prepared to know some basic sysadmin, or pay somebody to do it for you. My newest server has about 60 cores and half a tera of ram. Surprisingly, it's not uber sharp - I went with high core count so individual queries actually got slower for about 20%. But that load... you can't even tell if the cpu load gauge is working. I can't wait to fill it up :D Maybe this black friday season I'll get it to 10%.
If you do something stupid with your code at least you won't go bankrupt, only your service will be slower.
How can DO and Vultr even compete? Probably on the basis of their nicer dashboards and easier sign-up flow (especially OVH's)
Still, I'd not run my RDBMS on an unmanaged, non replicated dedicated server and I'd not bother setting up multiple servers with failover, automated backups etc and keep updating them. Fuck that, I'll pay whatever AWS says RDS costs.
AFAIK AWS doesn't have that. They do have the ability to send me alerts if my bill is unexpectedly high, but they still keep working until I go bankrupt. It's possible to use those alerts to implement your own "broke man's switch", but they don't have it built in.
In fact, it is slightly cheaper at AWS.
Ondemand PostgreSQL, Single Node, 1vCPU, 1GB MEM, 10GB Storage is $15 at DO
Ondemand PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $14.29 at AWS (db.t3.micro at us-east-2)
if reserved for 1yr no upfront
Reserved PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $10.57 at AWS (db.t3.micro at us-east-2)
Or you can use ARM and go lower.
Ondemand PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $12.83 at AWS (db.t4g.micro at us-east-2)
One insert every 3 seconds. Could run that off a 10 year old laptop.
Amazon RDS lowest-tier runs about $13/mo for 10GB storage, 2 vCPUs and 1GB memory with automated backups and push-button restoring. And that would have likely met all of their needs with capacity to spare.
This model also scales DOWN really well .. while still providing good scalable availability...
That said, I DO agree with the sentiment of paying for a set performance level (clu, memory, storage), to provide predictable pricing.. obviously these guys were bit by the scaling capability.
I do a lot of pet projects, and I find DynamoDB works really well because my pet projects cost $0 most months... And I don't have to worry about servers, maintenance, or what not... I'm happy to do that at work, but I don't want that for my friends & fun projects... And I've not seen a decent DB managed RDS for <$5/month
This is why BigQuery offers both models and lets you control the caps [1].
Buying fixed compute is effectively buying a throughput cap. Hard Quotas provide a similar function, but aren't a useful budgeting tool if you can't set them yourself.
"Serverless" without limits is basically "infinite throughput, infinite budget" (though App Engine had quotas since day 1 and then budgets once charging was added). The default quotas give you some of that budget / throughput capping, but again if you can't lower them they might not help you.
Either way, BQ won't drop ingestion or storage though because almost nobody wants their data deleted. As a provider, implementing strict budgets is impossible without having a fairly complex policy "if over $X/second stop all activity, oh except let me still do admin work, like adding indexes? Over $Y/second delete everything". I think having user adjustable quotas and throughput caps per "dimension" makes more sense but it puts the burden on the user and no provider offers good enough user control over quota.
tl;dr: true budgets are hard to do, but every provider should strive to offer better quota/throughput controls.
[1] https://cloud.google.com/bigquery/pricing
[2] https://cloud.google.com/bigquery/docs/reservations-workload...
It also seems very expensive compared to just renting DBs by instance, if you put any real load onto this. I can see this being attractive if your use case only queries single rows by key, but it's essentially a big minefield for any query more complex than that. A database with a rather opaque query planner doesn't seem like a good fit for this kind of pricing.
have you forgot that MongoDB is web scale?
Nice!
Even given the limitations of the time (RDBMS cost, 9GB disks, Sun kit, etc), our cost of good sold for that type of workload was exponentially less. (At scale) Today, I could probably run that company off my MacBook Pro and have room to spare.
That said, the rationale for choosing this technology is cute: “After seeing a ton of the best GitHub engineers end up at PlanetScale and seeing the process GitHub went through to issue simple migrations, we chose to use their service.”
If you use the same methodology to choose a database that the public uses to choose between Bud Light, Miller Lite, and Coors Light, expect a suboptimal outcome.
It is funny to look back, but getting huge bills without even realizing that we’re doing something wrong is very real possibility. Cloud vendors happily make their pricing opaque as it benefits them.
I’d avoid even the best product in the industry, if their pricing is opaque. Or if there is a “Contact Us” button when there needn’t be.
Or 1 query = 3 gb / snapshot storage for a month.
That is such a bad habit like everyone getting on git and getting burned and now it's irreversible with all the existing ecosystem.
How hard is it to just spin up a beefy cloud instance and run a MySQL of your own with whatever backup strategy you got and do things the way it is than getting bitten by using stuff you're not even familiar with.
Then a bro showed me one night about the magic of indexes. 5 minutes worth of advice saved me hundreds of dollars per month in the future and all he asked in return was for some beer and chicken wings.
Now that is a good bro.
I'm happy to say I've paid it forward myself.
Imagine how many wasted $$$ they earned based on common knowledge that they should prevent for customers instead.
My company in boarded `fivetran` to source data from different tools.
Budget got exhausted in sourcing `iterable` data
Interestingly enough, some DBs (like Cassandra) would refuse scan-type queries unless specifically asked to. I wonder if cloud-based DBs which charge per row inspected could have such mode... Though of course it's their incentive not to.
