> 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches within the affected Availability Zone. Connectivity and power to other data centers within the affected Availability Zone, or other Availability Zones within the US-EAST-1 Region are not affected by this issue, but we would recommend failing away from the affected Availability Zone (USE1-AZ4) if you are able to do so. We continue to work to address the issue and restore power within the affected data center.
So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain functions", so I guess it's not enough to handle full load while generators spin up if the 2 primaries go out. Or perhaps some failure in the source switching equipment (typically called a "static transfer switch").
Some detail on different approaches: https://www.donwil.com/wp-content/uploads/white-papers/Using...
If they claim tier4, then they basically have everything in n+n configuration.
I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated which one was hot). A major power outage hit, and there was a thunderous roar as all of the generators fired up. The channel never went off the air.
I wonder if a lot of AWS dc design in this area predates the battery grid storage revolution with (what my impression is) a far faster adaptation/switchover time than a generator spin up, and possibly software systems that work to detect and switch over quickly?
AWS can claim it will be best of breed, but they aren't going to throw out a DC power redundancy investment (or threaten downtime) that they can't wring more ROI on.
We're still waiting on the RCA for last week's us-west outage...
"almost unusable" is maybe exaggerating, but there were definitely issues affecting more than just the single AZ.
Yes, some of these we should be better at handling ourselves, but... it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands.
edit: just to short circuit any "well, why aren't you running redundant regions" - we run redundant regions at all times. But for reasons of latency, many customers will bind to their closest region, and the nature of our technology is highly location-bound It is not possible for us to move active sessions to an alternate region. So something like this is... unpleasant.
Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails?
IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between?
Deploying to multiple places is more expensive, it's not wrong to choose not to, it's trading off reliability for cost.
It's also unclear to me how often things fail in a way that actually only affect one AZ, but I haven't seen any good statistics either way on that one.
It's turtles all the way down, and underneath all the turtles is us-east-1.
Off the top of my head, US-EAST-1 is:
(1) topologically closer to certain customers than other regions (this applies to all regions for different customers),
(2) consistently in the first set of regions to get new features,
(3) usually in the lowest price tier for features whose pricing varies by region,
(4) where certain global (notionally region agnostic) services are effectively hosted and certain interactions with them in region-specific services need to be done.
#4 is a unique feature of US-East-1, #2-#3 are factors in region selection that can also favor other regions, e.g., for users in the West US, US-West-2 beats US-West-1 on them, and is why some users topologically closer to US-West-1 favor US-West-2.
I'm not sure why this is a big deal though, this is why Amazon has multiple AZ's. If your in one AZ, you take your chances.
IMO it was handled rather well and fast by AWS... not saying we shouldn't beat them up (for a discount) but being honest this wasn't that bad.
I once had to prepare for a total blackout scenario in a datacenter because there was a fault in the power supply system that required bypassing major systems to fix. Had some mistake or fault happened during those critical moments, all power would've been lost.
Well-designed redundancy makes high-impact incidents less likely, but you're not immune to Murphy's law.
It's turtles all the way down.
Interesting quote:
“This is exactly the sort of design that lets me sleep like a baby,” said DeSantis. “And indeed, this new design is getting even better availability” – better than “seven nines” or 99.99999 percent uptime, DeSantis said.
1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power in them.
I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time.
However, a power cut (in the traditional sense where the supplier has a failure so nothing comes in over the wire) should have literally zero effect!
What am I missing?
I've never worked anywhere with Amazon's budget so why are they not handling this? Is it more than just the imcoming supply being down?
Nothing happens if you remember that your new capacity limit per DC supply is 50% of the actual limit, and you're 100% confident that either of your supplies can seamlessly handle their load suddenly increasing by 100%.
I've seen more than one failure in a DC where they wired it up as you described, had a whole power side fail, followed by the other side promptly also failing because it couldn't handle the sudden new load placed on it.
Normally this is factored into the Rack you buy from a hardware provider, they will tell you that you have 10A or 16A on each feed, if you exceed that: it will work, but you are overloading their feed and they might complain about it.
This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale.
Data centers lose supply lines. They lose transformers. Sometimes they lose primary feed and secondary feed at the same time. Automatic transfer switches cannot be tested periodically i.e. they are typically tested once. Testing them is not "fire up a generator and see if we can draw from it"
It is cheaper to design a system that must be up which accounts for a data center being totally down and a portion of the system being totally unavailable than to add more datacenter mitigations.
And to top it off each rack had its own smaller UPS at the bottom and top, fed off both rails, and each server was fed from both.
We never had a power issue there; in fact SDGE would ask them to throw to the generators during potential brown-out conditions.
Of course this was a datacenter that was a former General Atomics setup iirc ...
Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in?
Again, this is all pure speculation: I have absolutely no idea of the exact failure, nor how their infrastructure is held together - this is all just speculation for the hell of it :)
That's ATS. It is not really advisable to test their under load performance because the failure of an ATS would be catastrophic. ATS typically would be tested at the installation and after that their parameters would be monitored.
Replacing a functional in line ATS would be a 9-12 months long project.
> Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in?
At high energy you are pretty much always going to use an ATS.
I've had bad power supplies fry out taking the whole power circuit with it, and thus half (or whatever fraction) of the rack's power. I've also had bad power supplies bring down the whole machine as they shunted everything internal too.
When things go bad, anything can happen. You can provide the best effort, and it'll usually work as expected, but there will always be something that can overcome your best efforts.
Citation needed - the same issue with testing, data races and expensive bandwidth come up.
For big DC workloads, it is usually, though not always, better to take the higher failure rate than add redundancy.
Actually, now that I type that it makes sense. Scaling a few tens of dollars to a bajillion servers on the off-chance that you get an inbound power failure (quite rare I'd reckon) might cost more than what they'd lose if it does actually fail.
So yeah, they're potentially just balancing the risk here and minimising cost on the hardware.
Edit: changed grammar a bit.
Perhaps we are going to discover how AWS produces such lofty margins by way of their next RCA publication.
My guess is that they cheaped out in having redundant PSUs to get you to use multiple availability zones. (More zones = more revenue)
Even a single PSU shouldn’t be an issue if they plugged in an ATS switch though.
Somewhat surprising to see how many things are failing though, which implies, either that a lot of services aren't able to fail-over to a different availability zone, or there is something else going wrong.
Specifically large parts of the management API, and IAM service are seemingly centrally hosted in us-east-1. So called Global endpoints are also dependent on us-east-1 and parts of AWS' internal event queues (eg. event bridge triggers)
If your infrastructure is static you'll largely avoid the fallout, but if you rely on API calls or dynamically created resources you can get caught in the blast regardless of region
The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for similar reasons. Always felt like a brittle house of cards to me (N.B. I do still use aws lambda, but keep it constrained to non-critical workloads).
One incident I recall was involving our GCP regional storage buckets, which we were using to achieve mutli-region redundancy. One day, both regions went down simultaneously. Google told us that the data was safe but the control plane and API for the service is global. Now I always wonder when I read about MR what that actually means...
Your point here deserves highlighting. A failure such as a zone failing is nowadays a relatively simple problem to have. But cloud services do have bugs, internal limits or partial failures that are much more complex. They often require support assistance, which is where the expertise of their staff comes into play. Having a single provider that you know well and trust is better than having multiple providers where you need to keep track of disparate issues.
You really need multi-region and also not be relying on any AWS service that’s located only in us-east-1 (including everything from creating new S3 buckets to IAM’s STS).
edit: The question isn't necessarily AWS specific, just any data on amount of downtime per cloud provider on a timeline would be nice.
There indeed has been an uptick in AWS outages recently. You can see a bit of the history here: https://statusgator.com/services/amazon-web-services
If you can’t even admit you’re having an issue how can you keep an accurate record?
Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZs, typically by just dropping nodes in the down AZ.
People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down.
Isn't that because Elasticache will distribute the cluster across AZs automatically?
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...
Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well.
And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything.
It's AWS job to ensure that things are reliable, that there is redundancy and that multi-AZ infra should be safe enough. The amount of issues in US-EAST-1 lately is really worrying.
I do agree that the end of this year has been a very bad period for AWS. I wonder whether there’s a connection to the pandemic conditions and the current job market – it feels like a lot of teams are running without much slack capacity, which could lead to both mistakes and longer recovery times.
In the past I've seen both of those systems seamlessly handle an AZ failure. Today was different.
Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. These entire datacenter failures should be anticipated, but to expect them is a bit too easy I think. There are plenty of hosters who don't have this stuff even once for the last decade in their only datacenter. I do not find it strange to expect or even demand that level but to protect yourself if it happens in any case if that fits your specific project and budget.
Edit; OK meant that raid-5 remark in the same context as the hosting; it can and does happen but it shouldn't; you should plan for a contingency but expect it goes far. We never had it (1000s of hard-drive, decades of hosting, millions of sites) and so we plan for it with backups; if it happens it will take some downtime but it costs next to nothing over time to do that. If we expected it, we would need to take far different measures. And we had less downtime in a decade than aws AZ had in the past months. I have a problem with the word 'expect'.
There are plenty of situations where this might happen if they’re in your room: a lightning strike can cause a surge that causes the disks to fry, a thief might break in and steal your system, your house might burn down, an earthquake could cause your disks to crash, a flood could destroy the machines, and a sinkhole could open up and swallow your house. You may laugh at some of these as being improbable, but I have seen _all_ of these take out systems between my times in Florida (lightning, thief, sinkhole, and flood) and California (earthquake and house fire).
The fix for this is the same fix as being proposed by the parent post - putting physical space between the two systems so if one place become unavailable you still have a backup.
Here are some examples where that happened:
1. Drive manufacturer had a hardware issue affecting a certain production batch, causing failures pretty reliably after a certain number of power-on hours. A friend learned the hard way that his request to have mixed drives in his RAID array wasn’t followed.
2. AC issues showed a problem with airflow, causing one row to get enough warmer that faults were happening faster than the RAID rebuild time.
3. UPS took out a couple racks by cycling power off and on repeatedly until the hardware failed.
No, these aren’t common but they were very hard to recover from because even if some of the drives were usable you couldn’t trust them. One interesting dynamic of the public clouds are that you tend to have better bounds on the maximum outage duration, which is an interesting trade off compared to several incidents I’ve seen where the downtime stretched into weeks due to replacement delays or manual rebuild processes.
Same manufacturer, same disk space, same location, same operator, same maintenance schedule, same legal jurisdiction, same planet, you name it, and there's a common failure to match
HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner.
All drives were purchased at the same time, and happened to all come from a single batch from the manufacture. The manufacture confirmed this via serial numbers, and admitted they had an issue during production. All drives were replaced and at a larger volume size.
TL;DR: Drives will fail, and manufacturing issues happend. Don't buy all of your drives in an array from the same batch! It will happen. To say it won't is just pure inexeperience.
The context of the parent seems to be that they intermittently couldn't get to the console. That seems fair to me. If we're blaming developers and finding gaps in HA design, then AWS should also figure out how to make the console url resilient. If it's not, then AWS does appear to be down.
I imagine it's pretty hard to design around these failures, because it's not always clear what to do. You would think, for example, that load balancers would work properly during this outage. They aren't. Or that you could deploy an Elasticache cluster to the remaining AZs. You can't. And I imagine the problems vary based on the AWS outage type.
Similarly, with the earlier broad us-east-1 outage, you couldn't update Route53 records. I don't think that was known beforehand by everyone that uses AWS. You can imagine changing DNS records might be useful during an outage.
As others have said, they are not being forthright about the severity of the issue, as is standard.
edit: of course, AWS does have this: AWS Fault Injection Simulator
Be in multiple AZs, and even multiple regions but if you're going to be in only one AZ or one region, make it us-east-2.
I have a server at OVH (not affiliated to them) which, at this point, I keep only for fun. It has 3162 days of uptime as I type this.
3 162 days. That's 8 years+ of uptime.
Does it have the traffic of Amazon? No.
Is it secure? Very likely not: it's running an old Debian version (Debian 7, which came out in, well, 2013).
It only has one port opened though, SSH. And with quite a hardened SSH setup at that.
I installed all the security patches I could install without rebooting it (so, yes, I know, this means I didn't install all the security patches for some required rebooting).
This server is, by now, a statement. It's about how stable Linux can be. It's about how amazingly stable Debian is. It's also about OVH: at times they had part of their datacenter burn (yup), at times they had full racks that had to be moved/disconnected. But somehow my server never got affected. It may have happened that at one point OVH had connectivity issues but my server went down.
I "gave back" many of my servers I didn't need anymore. But this one I keep just because...
I still use it, but only as an additional online/off-site backup where I send encrypted backups. It's not as if it gets zero use: I typically push backups to it daily.
They're only backups, they're encrypted. Even if my server is "owned" by some bad guys, the damage he could do is limited. Never seen anything suspicious on it though.
I like to do "silly" stuff like that. Like that one time I solve LCS35 by computing for about four years on commodity hardware at home.
I think it's about time I start to do some archeology on that server, to see what I can find. Apparently I installed Debian 7 on it in mid-october 2013.
I've created a temporary user account on it, which at times I've handle the password (before resetting it) to people just so they could SSH in and type: "uptime".
It is a thing of beauty.
Eight. Years. Of. Uptime.
Awesome! Are you Bernard Fabrot [0]?
[0] https://www.csail.mit.edu/news/programmers-solve-mits-20-yea...
At its current use, it's likely not a major issue but imagine if someone saw this uptime and thought to take it as a statement of reliability and built a service on it. I for one, would want that disclosed because this is a disaster waiting to happen. I'd much rather someone disclose that they had a few servers each with no longer than 7 days of uptime because they'd been fully imaged and cycled in that time...
Simiarly, my laptop, if I keep it plugged in the wall, and enable httpd on localhost, will surely have better uptime than any of the top clouds. I'd bet that it'd have 100% uptime if I plugged in a UPS and cared for traffic on my local network only.
In reality, most people don't need to scale. An occasional spike in traffic is a nuisance, but not the end of the world, and security is not terribly hard, if you keep your servers patched (which is trivial to automate).
I really don't understand why there's so much FUD around running your own stuff.
When your server requirements get into needing 5-6 servers (not at all atypical for a startup in their first year of being launched), running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount.
Of course it doesn't. Why are you asking antagonistic questions?
Is this not antagonistic? It's pointless to make these statements, so your parent comment pointed it out. Go downvote the first one instead.
Better uptime than paying for EC2 on AWS US-East-1.
Obviously this approach isn't scalable but it serves me well.
It's perfectly scalable. Just give everybody their own home server.
“ditch your on-prem infrastructure and migrate to a major cloud provider”
And its starting to seem like it could be something like:
“ditch your on-prem infrastructure and spin up your own managed cloud”
This is probably untenable for larger orgs where convenience gets the blank check treatment, but for smaller operations that can’t realize that value at scale and are spooked by these outages, what are the alternatives?
A much faster and more effective solution that doesn't have you trading cloud problems with on-prem problems (the power outage still happens, except now it's your team that has to handle it) would be to update your services to run in multiple AZs and multiple regions.
Get out of AWS is you want, but don't get out of AWS because of outages. You should be able to mitigate this relatively easily.
Everything fails, we can argue the rate. But I would argue that understanding your constraints is better.
if you know that your secret storage system can't survive if a machine goes away: well, you wire redundant paths to the hardware and do memory mirroring and RAID the hell out of the disks. And if it fails you have a standby in place.
But if you use AWS Cognito.
And it goes down.
You're fucked mate.
I remember we had a power outage in 2006, it actually took one of my services off air. Since then of course that has been rectified, and the loss of a building wouldn't impact on any of the critical, essential or important services I provide.
Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the cloud ones?
And I'm not talking about "we have 12 SREs who are based in Cupertino and are all paid top dollar to support a colo"...I'm talking average.
And what rate is this? It gets attention because it impacts more people, but AWS / GCP / Azure uptime is still better than what I've seen for small / mid size businesses trying to manage their own infrastructure.
Commenters will show up like clockwork and say shit like:
“What man, it’s not like cars didn’t crash before? Haha”
Don’t be dense dude. And definitely don’t pursue a leadership position anytime in the future.
The problem with both this example, and the AWS one (it needs to have better availability than your personal home-spun solution, and it does), is that people are amazing at deluding themselves.
"Yes, cars are dangerous, because other people can't drive. But I'm a better than average driver"
"Yes, other people will build unreliable systems. But I know how to architect for my use case and ensure that for my needs the availability will be higher than AWS's"
Both are true* in the micro sense and false in the macro sense.
* Not really. 88% of americans think they are "above average" drivers.
If your app depends on a few 3rd party services -- SendGrid, Twilio, Okta and they're all hosted on different infra then congrats! You're gonna have issues when any one of them are down, yayyy.
Also the marketing benefit can't be downplayed. If your postmortem is "AWS was having issues" then your execs and customers just accept that as the cost of doing business because there's a built-in assumption that AWS, Azure, GCP are world class and any in-house team couldn't do it better.
Edit: Restarting Slack does update the edited messages.
Edit 15:24 CET: Slack is back up.
- edits failing or working with big lag;
- "Threads" view slow;
- can't emoji-react;
- can't upload images;
- people also say they can't join new channels.
> We are experiencing issues with file uploads, message editing, and other services. We're currently investigating the issue and will provide a status update once we have more information.
> Dec 22, 1:58 PM GMT+1
remote: Compressing source files... done.
remote: Building source:
remote:
remote: ! Heroku Git error, please try again shortly.
remote: ! See http://status.heroku.com for current Heroku platform status.
remote: ! If the problem persists, please open a ticket
remote: ! on https://help.heroku.com/tickets/newAnother thread: https://news.ycombinator.com/item?id=29648325
In all seriousness though - even non-regional AWS services seem to have ties to us-east-1 as evidenced by the recent outages. So you might be impacted even if it looks like (on paper at least) you’re not using any services tied to that region.
The console is throwing errors from time to time. As usual no information on AWS status page.
aws ec2 describe-availability-zones | jq -r '.AvailabilityZones[] | select(.ZoneId == "use1-az4") | .ZoneName'https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#...:
(Edit: I hope I didn't sound sarcastic. I don't open random console pages and scroll all the way down to check for new features. Some people will have noticed, some won't.)
The letters are randomised per AWS account so that instances are spread evenly and biases to certain letters don't lead to biases to certain zones.
The thing is, us-east-1 represents the whole AWS for the majority of us.
I am not sure if the movement the cloud has reduced amount of failures, but it definitely has made these failures more catastrophic.
Our profession is busy makin the world less reliable and more fragile, we will have our reconning just like the shipping industry did.
Today, on Slack i could not edit messages, could not edit statuses and could not post attachments. Pretty annoying!
Packaging a way to migrate off AWS could be a unicorn idea.
I mean I'm glad it exists, don't get me wrong. Just weird that they'd have two status pages, one seemingly existing only to sort of 'mock' themselves...
It's just strange.
Apparently it does some simple transformations of the actual status page, which is why the Amazon copyright stuff is in there.
He also has a decent newsletter and witty commentary, for all things AWS.
[0] https://twitter.com/quinnypig/status/1468331194471178241?s=2...
[05:01 AM PST] We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches within the affected Availability Zone. Connectivity and power to other data centers within the affected Availability Zone, or other Availability Zones within the US-EAST-1 Region are not affected by this issue, but we would recommend failing away from the affected Availability Zone (USE1-AZ4) if you are able to do so. We continue to work to address the issue and restore power within the affected data center.
Like how come down detector can do a superb job of detecting when AWS goes down and AWS can't? Because AWS doesn't want account managers of SLAs asking for credits for the uptime they're paying for but not getting.
That, or people never took the “if AWS goes down then lots of people will have a problem, so we’ll be fine” line seriously; there are few such cases.
Edit: Not supporting amazon, i generally dislike the company. I just don't understand the extend to which the criticism is justified
1. Did AMZN build an appropriate architecture?
2. Did AMZN properly represent that architecture in both documentation and sales efforts?
3. What the heck is going on with AMZN?
Let's say that they build an environment in which power is not fully redundant and tested at the rack level, but is fully redundant and tested across multiple availability zones. Did they then issue statements of reliability to their prospective and existing customers saying that a single availability zone does not have redundant power, and customers must duplicate functionality in at least 2 AZs to survive a SPOF?
Notably: cognito, r53 and the default web UI. (You can work around the webui one I’m told, by passing a different domain instead of just console.aws.amazon.com)
just the weekly internet apocalypse, happy holdidays fellow SREs
I'm having issues with Slack from central EU (Poland) -- can't upload images, or send emoji reactions to post; curiously, text works fine). Wondering if linked
Only with AWS and Github do I seem get panicked text messages on my phone first thing in the morning... Our workloads on Azure typically only have faults when everyone is in bed.
Santa is bringing me a Synology in three days.
> Due to this degradation your instance could already be unreachable
>:(
So it's not shocking to me that something going down in us-east-1 could have impact on other regions.
Meanwhile, I currently have a gig to work on a video service which features a never updated centos 6, an unsupported python 2 blob website, and a push to prod deployment procedure, running a single postgres db serving streaming for 4 millions users a month.
And it's got years of up time, cost 1/100th of AWS, and can be maintained by one dev.
Not saying "cloud is bad", but we got to stop screaming old techs are no good either.
2 - Caching is life. We have 3 layers of caching: cloudflare, varnish, and redis. Most things don't need to be real time. A lot of things can be a month old and the user doesn't care. User need immediate feedback to be happy, but not necessary fresh data.
3 - if you compile nginx manually, you get to use a lot of plugins that can do stuff super fast, including serving videos. You can script stuff in lua that will just skip the backend completly.
4 - mind your encoding. We carefully chose how we encode videos. The ffmpeg parameters are pretty insane, but the space / quality ratio is amazing, espacially on mobile. It takes a lot of time to experiment with those, nobody share them :)
5 - we offload everything we can to cron tasks or task queues. Including, obviously, encoding, screenshooting, etc.
6 - don't hold data you can't lose. E.G: billing. This way you can have a relaxed attitute toward data. If we ever loose a day of business, users will be in a bad mood for a week, but that won't be the end of the world. We don't need a bullet proof system if bullets can't kill us.
7 - give money to ffmpeg and opencv, because damn those things are fast. And good.
8 - servers are hosted accross 2 providers. This way, if one goes down, or decide to stop doing business with us Google style, we have a second one. Happened recently with leaseweb: they shutdown a whole room without offering an alternative.
E.G: votes.
They don't hit the backend on write. We pile them from nginx to redis, then once a day, we aggregate and store on postgres, which the backends will consumme. We just store each vote on localstorage as well so that the user feels like it's real time when they vote, but in reality it's updated once a day. But votes don't affect the money side of our business, so if we lose them one day, it does not mean death.
P.S: yes, posgres/redis/elasticsearch only hold metadata. Videos are stored on disk. There is no docker images, no mircoservices, FS is ext4. Which means with a lot of RAM, the OS FS cache will have most popular videos already loaded and ready to be streamed. Everything is raid 0, so if we get one disk corrupted, you lose the server. But we upload each videos on severeal servers, so when a disk get corrupted, we just replace the whole server. In fact, anything goes wrong on a server, we replace it. It's not worth it to find the root cause, unless 2 servers die in the same way successively.
At least, that's what I understood.
Another side benefit of being with AWS is when you do have an outage, a lot of other people have outages, and so you sort of blend in with the noise. It's not great to be down, but if you're down and also "big service X" who's also an AWS customer is down, it makes your downtime look less like a lack of competence and more like an unavoidable force of nature.
Though I guess there's still probably just lost revenue that could be captured by having better uptime, even if your competitors are down.
Agreed. Arguably, not using an existing cloud service is a red flag on any new hires. AWS being the primary, but experience using GCS or Azure are at least viable skills, even if your business is AWS-based.
But the "fad-based-development" meme is not going away any time soon. The incentives in the business are built around it (really! No one want's to work on a boring old relational database solution any more). In the old days it was 4th generation languages, RUP, XML and Function Point Analysis... today it's functional programming, SDKs, big-three cloud PaaS experience or (shudder) block-chain.
I think back to my much younger self, when I thought that technology was something to be mastered to solve real-world problems, and I laugh. Little did I know the real problem to be solved was to figure out how to solve those same-old business problems but with the technology of the season (Kubernetes, GraphQL or ML).
"Let's move our internal app with 50 users to k8s in the cloud." --true story
Uptime improved rather dramatically after that.
When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it.
Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region
AWS consists of over 200 services offered in 86 availability zones in 26 regions each with their own availability.
If one service in one availability zone being impaired equals a post about “AWS is down” we might as well auto-post that every day.
Specifically large parts of the management API, and IAM service are seemingly centrally hosted in us-east-1.
If your infrastructure is static you'll largely avoid the fallout, but if you rely on API calls or dynamically created resources you can get caught in the blast regardless of region
Wonder if it's connected?
Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" <-Quite literally their words.
Me: leaves cause that treatment was the final straw
Amazon and Facebook both go down within a month of each other, and supposedly they needed backups
Them: shocked pikachu face
I must admit that I do always try and maintain a separate data backup for true disaster recovery scenarios - but those are mainly focused around AWS locking me out of our AWS account (and hence we can't access our data or backups) or recovering from a crypto scam hack that also corrupts on-platform backups, for example.
What happens if AWS or [insert other megacloud] decides your account needs to be nuked from orbit due to a hack or some other confusion? We almost had this happen over the summer because of a problem with our bank's ability to process ACH payments. Very frustrating experience. Still isn't fully resolved.
What happens if an admin account is taken over and your account gets screwed up?
What happens if an admin loses his shit and blows up your account?
What happens if your software has a bug that destroys a bunch of your data or fubars your account?
There's a ton of cases where having at least a simple replica of your S3 buckets into a third-party cloud could prove highly valuable.
Was it just a miscommunication around AWS billing and them thinking you weren't paying? Or did AWS somehow put itself in the middle of, or react to, your use of ACH payment processing for *non-AWS* receivables or payables?
If the latter, that's a business risk I'd never even thought about. I'm not even sure how they'd know. But I'm thoughtful that things like the MATCH list [0] exist, and how easily a merchant can accidentally wind up on these lists from either human error or a small amount of high-value chargebacks. If cloud providers are somehow paying attention to merchant services reputation, that would be very scary for many businesses!
[0] https://www.merchantmaverick.com/learning-terminated-merchan...
1) Can you make your on prem infrastructure go down less than Amazon's?
2) Is it worth it?
In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted.
--
EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on a spectrum of possible compromises between cost and reliability. It might not be right for you. If it is too expensive -- go for cheaper options with less reliability.
If it is too unreliable -- go build your own yourself, but make sure you are not making huge mistake because you may not understand what it actually costs to build to AWSs level.
For example, personally, not having to focus on infra reliability makes it possible for me to focus on other things that are more important to my company. Do I care about outages? Of course I do, but I understand doing this better than AWS has would cost me huge amount of focus on something that is not core goal of what we are doing. I would rather spend that time thinking how to hire/retain better people and how to make my product better.
And adding all that complexity of running this infra to my company would cause entire organisation be less flexible, which is also a cost.
So you can't look at cost of running the infra like a bill of materials for parts and services.
And if there is an outage it is good to know there is huge organisation there trying to fix it while my small organisation can focus preparing for what to do when it comes back up.
Obviously depends on what you need, but for a small to medium web app that needs a load-balancer, a few app servers, a database and a cache, yes absolutely - all of these have been solved problems for over a decade and aren't rocket science to install & maintain.
> Is it worth it?
I'd argue that the "worth" would be less about immunity to occasional outages but the continuous savings when it comes to price per performance & not having to pay for bandwidth.
> overestimate how important it is for their services to run uninterrupted.
Agreed. However when running on-prem, should your service go down and you need it back up, you can do something about it. With the cloud, you have no choice but to wait.
You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage.
You need a back up generator and to be a short distance away from a petrol station so you can refuel quickly and regularly when suffering from longer durations of power outages. You absolutely do not want to run out of diesel!
You need redundancy of every piece of hardware AND you need to test that failover works as expected because the last thing you need is a core switch to fail and traffic not to route over secondary core switch like expected.
You need your multiple air con units and them to be powered off different mains inputs so if the electrics fail on one unit it doesn’t take out the others. I guarantee you that if the air cons will fail, it will be on the hottest day of the year a month amount of portable units will stop your servers from overheating.
You need beefy UPS with multiple batteries. Ideally multiple UPSs with each UPS powering a different rail on your racks so that if one UPS fails your hardware is still powered from the other rail. And you need to regularly check the battery status and loads on the UPS. Remember that the back up generator takes a second or two to kick in so you need something to keep the power to the servers and networking hardware to be uninterrupted. And since all your hardware is powered via the UPS, if that dies you still lose power even if the building is powered.
And you then need to duplicate all of the above in second location just in case the first location still goes down.
By the way, all of the possible failure points I’ve raised above HAVE failed on me when managing HA on prem.
The reason people move to the cloud for HA is because rolling your own is like rolling your own encryption: it’s hard, error prone, expensive, and even when you have the right people on the team there’s still a good chance you’ll fuck it up. AWS, for all its faults, does make this side of the job easier.
If you're outsourcing that, you'd likely have to pay a boatload just for someone to be available for help, let alone the actual tasks themselves. Like you said, if you're on-prem and something goes down, you can do something. But you've gotta have the personnel to actually do something.
That said, I think you're spot-on as long as you have the skillset already.
Backup is cheap when you're focused about what you're backing up.
In this case, the game isn't "going down less than Amazon", it's about going down uncorrelated to Amazon. Though that's getting harder!
"In more than one way" doesn't have to be local, but it may be across multiple cloud services. Still, "local" is nice in that it doesn't require the Internet. ("The Internet" doesn't tend to go down, but the portion you are on certainly can.) Of course, as workers disperse, "local" means less and less nowadays.
Over the last two years, my track record has destroyed AWS. I've got a single Mac Mini with two VMs on it, plugged in to a UPS with enough power to keep it running for about three hours. It's never had a second of unplanned downtime.
About 15 years ago I got sick of maintaining my own stuff. I stopped building Linux desktops and bought an Apple laptop. I moved my email, calendars, contacts, chat, photos, etc, to Google. But lately I've swung 180 degrees and have been undoing all those decisions. It's not as much of a PITA as I remember. Maybe I'm better at it now? Or maybe it will become a PITA and I'll swing right back.
EDIT: I realize you're talking in a commercial sense and I'm talking about a homelab sense. Still, take my anecdote for what it's worth. :D
I think you're right about what we over- & under-estimate, but that we also under-estimate the inflection point for when it makes sense to begin relying on major cloud services. Put another way: we over-estimate our requirements, causing us to pessimistically reach for services that have problems that we'd otherwise never have.
For extra safety, and extra work, you could even take Azure as a backup if you're not locked in with AWS.
Global services such as route53, Cognito, the default cloud console and Cloudfront are managed out of US-East-1.
If us-east-1 is unavailable, as is commonly the case, and you depend on those systems, you are also down.
it does not matter if you're in timbuktu-1, you are dead in the water.
it is a myth that amazon availability zones are truly independent.
please stop blaming the victim, because you can do everything right and still fail if you are not aware of this; and you are perpetuating that unawareness.
It's now hard to say how frequently Amazon's infrastructure goes down. The incident rate seems to have accelerated.
...My home Internet even is scoring better than Amazon right now, in fact. Yours probably is too.
In my experience problem number 3 is the hardest to solve.
At AWS, we built a few layers of redundant infrastructure with mulit-AZ availability within a region and then global availability across multiple regions. All this was done at roughly half the cost of the traditional hosting, even when including the additional person-hours required to maintain it on our end.
Keeping our infra simple helped that work, and it's literally been years since an outage caused by any AWS issues, even though there have been several large AWS events.
AWS maintains a fiction of turnkey infrastructure, and the reality of building your own is so starkly different that I haven't seen an IT group for some time that could successfully push back on these sorts of discussions.
Building your own datacenter is still too much like maintaining a muscle car, fiddly bits and grease under your fingernails all the time, meanwhile the world has moved on, and we now have several options in soccer mom EVs that can challenge a classic Corvette in the quarter mile, and obliterate its 0-60-0 time. There is no Hyundai for the operations people, and there should be.
I don't know the physics of shipping such a thing, but I think we really do need to be able to buy a populated and pre-wired rack and slot it into the data center. Literally slot it in. If you've ever been curious about maritime shipping, you know that they have a system for securing containers to cranes, trailers, each other, and I don't see a reason you couldn't steal that same design for mounting a server rack to the floor. Other than the pins would need to be removable (eg, a bolt that screws into a threaded hole in the floor) so you don't trip on them.
In a word, we need to make physical servers fungible. There are any number of things that we need to do to get there, but I think we can. Honestly I'm surprised we haven't heard more of this sort of talk from Dell, especially after they bought VMWare. This just seems like a huge failure of imagination. Or maybe it's simply a revolution lacking a poster child. At this rate that 'child' has already been born, and we are just waiting to see who it is.
I'd wager that will still give you more uptime than a physically-hosted solution for the same cost.
It’s not a bad idea store backups offline but costs might make that an expensive proposition.
I've had buckets and objects disappear into the ether.
It is exceedingly rare, but it's not impossible.
Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit.
Most of the folks impacted by cloud outages do not have highly available systems in place. Perhaps, for their business, the cost doesn't justify the outcome.
If you need high uptime for instances, build your system to be highly available and leverage the fault domain constructs your provider offers (placement groups, availability zones, regions, load balancing, DNS routing, autoscaling groups, service discovery, etc). For instances, double down and use spot instance and maximum lifetimes in your groups so that you're continuously validating your application can recovery from instance interruptions.
If you're heavy on applications that leverage cloud APIs, such as is often the case with labmdas, then strongly consider multi-region active/active as API outages tend to cross AZ's and impact the entire region.
Maybe a cheery note asking how the team is doing, sent right in the middle of an outage.
Passive aggressive? As hell. Cathartic? Damn skippy.