1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN).
2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.
3) The sum total of information about this incident can be found as a few one or two sentence blurbs on Google's blog. No explanation nor outline of scope for affected regions and services has been provided.
4) Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems.
5) Some users here are reporting that they have received no response from GCP support, even over a time span of 40+ hours since the support request was submitted.
6) Google says they'll provide some information when the next business day rolls around, roughly 4 days after the start of the problem.
I really do want to make sure I'm understanding this situation. Please do correct me if I got something wrong in this summary.
When things stop working, GCP is the worst. Slow communications and they require way too much work before escalating issues or attempting to find a solution.
They already have the tools and access so most issues should take minutes for them to gather diagnostics, but instead they keep sending tickets back for "more info", inevitably followed by a hand-off to another team in a different time zone. We have spent days trying to convince them there was an issue before, which just seems unacceptable.
I can understand support costs but there should be a test (with all vendors) where I can officially certify that I know what I'm talking about and don't need to go through the "prove its actually a problem" phase every time.
The issue with outages for the Government organizations I have dealt with is rarely the outage itself - but strong communication about what is occurring and realistic approximate ETAs, or options around mitigation.
Being able to tell the Directors/Senior managers that issues have been "escalated" and providing regular updates are critical.
If all I could say was a "support ticket" was logged, and we are waiting on a reply (hours later) - I guarantee the conversation after the outage is going to be about moving to another solution provider with strong SLAs.
When I worked at GoDaddy, there were around 2/3 of the company was customer support.
At the current company I'm at, a cryptocurrency exchange, our support agents frequently hear they prefer our service over others because of our fast support response times (crypto exchanges are notorious for really poor support).
All of my interactions with Amazon support have been resolved to my satisfaction within 10 minutes or less.
Companies really ought to do the math on the value that comes from providing fast, timely, and easy (don't have to fight with them) customer support.
Google hasn't learned this lesson.
Isn't that the case with basically every support request, no matter the company or severity? The first couple of emails from 1st & even 2nd level support are mostly about answering the same questions about the environment over and over again. We've had this ping-pong situation with production outages (which we eventually analysed and worked around by ourselves) and fairly small issues like requesting more information of an undocumented behavior which didn't even effect us much. No matter how important or urgent the initial issue was, eventually most requests end up being closed unresolved.
https://www.hanselman.com/blog/FizzBinTheTechnicalSupportSec...
So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had rampant impact on other services.
But i find it concerning that they provided very little update on what went wrong. I also think its better to expect nil support out of any big cloud provider if you don't have paid support. Funny how all these big cloud providers think you are not eligible for support de-facto. Sigh.
If you are an early stage startup can you afford their 200/Month support, when your entire GCP bill is under $1. However, that doesn't mean you don't have to support them.
"We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI."
So, it's a UI console issue, it appears you can still manage
"Affected customers can use gcloud command [1] in order to create new Node Pools. [1]"
Similarly, it actually was resolved in Friday, but they forgot to mark it as so.
"The issue with Google Kubernetes Engine Node Pool creation through the Cloud Console UI had been resolved as of Friday, 2018-11-09 14:30 US/Pacific."
The items I put down in my comment are based largely on user reports, though (there isn't much else to go on). And I mean these items as questions (i.e. "is this accurate?"). Folks here on HN have definitely been reporting ongoing problems and seem to be suggesting that they are not resolved and are actually larger in scope than the Google blog post addressed.
Someone from Google commented here a few hours ago indicating Google was looking into it. And other folks here are reporting that they don't have the same problems. So it's kind of an open question what's going on.
I'm in the evaluation phase too. And I've found a lot to like about GCP. I'm hoping the problems are understandable.
Edit: I finally got my cluster up and running by removing all nodes, letting it process for a few minutes, then adding new nodes.
I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend.
On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up.
And yet, the status page says all services are available.
What blog statement are you referring to? I don't see any such statement. Can you provide a link?
The OP incident status issue says "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI". It also says "Affected customers can use gcloud command in order to create new Node Pools."
So it sounds like a web interface problem, not a severely limiting, backend systems problem with global scope.
Also, the report says "The issue with Google Kubernetes Engine Node Pool creation through the Cloud Console UI had been resolved as of Friday, 2018-11-09 14:30 US/Pacific". So the whole issue lasted about 10 hours, not three whole days.
> Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems
I don't see much of that.
https://status.cloud.google.com/incident/container-engine/18...
"We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI."
> So it sounds like a web interface problem, not a severely limiting
Depends who you as to whether this is "severely" limiting, but yes there is a workaround by using an alternate interface.
a) Google had a global service disruption that impacted Kubernetes node pool creation and possible other services since Friday. They had a largely separate issue for a web UI disruption (what this thread links to) which they forgot to close on Friday. They still have not provided any issue tracker for the service distribution and it's possibly they only learned about it from this hacker news thread.
b) People are having various unrelated issues with services that they're mis-attributing to a global service disruption.
...and I'm a happy GCP customer.
Ok. So on aws we were* paying for putting systems across regions, but, honestly I don’t get the point. When an entire region is down what I have noticed is that all things are fucked globally on aws. Feel free to pay double - but it seems* if you are paying that much just pay for an additional cloud provider. Looks like it’s the same deal on GCP.
Do you have an example on this?
That being said I really do think there is a difference between who is working at google today and the google we all fell in love with pre-2008.
I am sure there are a amazing people still working at google, but nowhere near like it was.
The way I like to think about google is that some amazing people mad ea awesome train that builds tracks in front of it -- you can call them gods maybe -- but those people are gone -- or a least the critical mass required to build such a train has dwindled to just dust. What we have left is a awesome train full of people pulling the many levers left behind.
To make things even worse my last interview as a SRE left me wondering if even the people who are there know this as well, and they are actually working hard to keep out those who might expose light on to this. I don't say that because I did not get the job -- I am actually happy I did not get extended a offer.
I say this with one exception, the old-timer who was my last interview. I could tell he was dripping in knowledge and eager to share it with any that would listen. I came out of his 45 min session learning many things -- I wold actually pay to work with a guy like that.
I would also like to point out that the work ethic was not what I expected. I was told that when on call, my duty was to figure out the root cause was in the segment I was responsible for. I don't know about you, but if my phone rings at night I am going to see through to a resolution and understand the problem in full -- even if it is not on the segment that I was assigned.
/end rant
During my time on the GCE team (note I don't work at Google now) I knew multiple full-time Google employee support reps, including some still at the company. They have the good attitude and deep knowledge you'd hope for.
The problem is simply about how Google scales their GCP support org. To be completely clear, AWS support is by and large not great either.
If you're a big or strategically important customer, of course, you can get a good response from either company.
Perhaps if you explained it on a whiteboard...
From my personal experience - i think all big cloud providers first two level support staff is no good if it isn't an obvious dumb one on your part. I always prefer to forgo support and try to go through every bit of their documentation to figure it out on our own. This helps to save huge amount of time. But if you have developer support - it can help to expedite things little faster though.
That's my favorite.
- Someone at Google right now, probably.
(I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)
for any reason
at any time
It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.
As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the "cloud" was supposed to make this kind of failure impossible
I've been creating GCP instances in us-central1-a and us-central1-c today without issue. Which zone were you using in NA?
I have been noticing unusual restarts, but I haven't been able to pin down the cause yet (may be my software and not GCP itself).
You have to remember that you're trying to have access to backend platforms and infrastructure at all times, which almost no public utility does (assuming "the cloud" is "public utility computing"). Power plants go into partial shutdown, water treatment plants stop processing, etc. Utilities are only designed to provide constant reliability for the last mile.
If there's a problem with your power company, they can redirect power from another part of the grid to service customers. But some part of your power company is just... down. Luckily you have no need to operate on all parts of the grid at all times, so you don't notice it's down. But failure will still happen.
Your main concern should be the reliability of the last mile. Getting away from managing infrastructure yourself is the first step in that equation. AppEngine and FaaS should be the only computing resources you use, and only object storage and databases for managing data. This will get you closer to public utility-like computing.
But there's no way to get truly reliable computing today. We would all need to use edge computing, and that means leaning heavily on ISPs and content provider networks. Every cloud computing provider is looking into this right now, but considering who actually owns the last mile, I don't think we're going to see edge computing "take over" for at least a decade.
If set up properly to be utilized correctly, yeah. But, it's not a perfect world though.
People who respond here could be employees of Google, caring about it and respond here because they know it.
What he can mention ( a lot of people are working on it) is what you can suspect when something is going down. All other cloud providers do the same.
There is a reason while Google have been having hard time making inroads in the enterprise cloud. Kind of impedance mismatch between enterprise and the Google style. That 2 stories like high "We heart API" sign on the Google Enterprise building facing 237 just screams about it :)
I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend.
On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up.
And yet, the status page says all services are available.
Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich cloud (particularly your networking products), but down time like this is unacceptable.
The problem is that running a global SDN like this means if you do something wrong, you can have outages that impact multiple regions simultaneously.
This is why AWS has strict regional isolation and will never create cross-region dependencies (outside of some truly global services like IAM and Route 53 that have sufficient redundancy that they should (hopefully) never go down).
Disclaimer: I work for AWS, but my opinions are my own.
Disclaimer: google employee in ads, who worked on many many fires throughout the years, but talking from my personal perspective and not from my employer. I am sure we are striving to have 0, but realistically, i have seen many that says things happen. Learn, and improve.
(Disclosure: I worked for Google, including GCP, for a few years ending in 2015. I don't work or speak for them now and have no inside info on this outage.)
Most of what you can read of Google's approach will teach you their ideal computing environment is a single planetary resource, pushing any natural segmentation and partitioning out of view.
It's the opposite really: the expectation that service providers have no unexpected downtime is unrealistic, and it's strange this idea persists.
I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things:
1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumably given this is multiple days in to the issue, rolling back isn't an option. That doesn't say good things about their testing or deployment stories, and possibly their monitoring of the product? Even if the deployment validation processes failed to catch it, you'd really hope alarming would have caught it.
or:
2) Regions aren't isolated from each other. Cross-region dependencies are bad, for all sorts of obvious reasons.
5. Years.
Nothing to see here, move along.
No one would ever ask why you chose AWS. The old “no one ever got fired for buying IBM”.
Even if you chose Azure because you’re a Microsoft shop, no one would question your choice of MS. Besides, MS is known for thier enterprise support.
From a developer/architect standpoint, I’ve been focused the last year on learning everything I could about AWS and chose a company that fully embraced it. AWS experience is much more marketable than GCP. It’s more popular than Azure too, but there are plenty of MS shops around that are using Azure.
- Security posture. Project Zero is class leading, and there's absolutely a "fear-based" component there, with the open question of when Project Zero discovers a new exploit, who will they share it with before going public? The upcoming Security Command Center product looks miles ahead of the disparate and poorly integrated solutions AWS or Azure offers.
- Cost. Apples to apples, GCP is cheaper than any other cloud platform. Combine that with easy-to-use models like preemptible instances which can reduce costs further; deploying a similar strategy to AWS takes substantially more engineering effort.
- Class leading software talent. Google is proven to be on the forefront of new CS research, then pivoting that into products that software companies depend on; you can look all the way back to BigQuery, their AI work, or more recently in Spanner or Kubernetes.
- GKE. Its miles ahead of the competition. If you're on Kubernetes and its not on GKE, then you've got legacy reasons for being where you're at.
Plenty of great reasons. Reliability is just one factor in the equation, and GCP definitely isn't that far behind AWS. We have really short memories as humans, but too soon we seem to forget Azure's global outage just a couple months ago due to a weather issue at one datacenter, or AWS's massive us-east-1 S3 outage caused by a human incorrectly entering a command. Shit happens, and it's alright. As humans, we're all learning, and as long as we learn from this and we get better then that's what matters.
Or you have legitimate reasons for running on your own hardware, e.g. compliance or locality (I work at SAP's internal cloud and we have way more regions than the hyperscalers because our customers want to have their data stay in their own country).
But, whether it is right or not, as an architect/manager, etc, you have to think about what’s not just best technically. You also have to manage your reputational risks if things go south and less selfishly, how quickly can you find someone with the relevant experience.
From a reputation standpoint, even if AWS and GCP have the same reliability, no one will blame you if AWS goes down if you followed best practices. If a global outage of an AWS resource went down, you’re in the same boat as a ton of other people. If everyone else was up and running fine but you weren’t because you were on the distant third cloud provider, you don’t have as much coverage.
I went out on a limb and chose Hashicorp’s Nomad as the basis of a make or break my job project I was the Dev lead/architect for hoping like hell things didn’t go south and the first thing people were going to ask me is why I chose it. No one had heard of Nomad but I needed a “distributed cron” type system that could run anything and it was on prem. It was the right decision but I took a chance.
From a staffing standpoint, you can throw a brick and hit someone who at least thinks they know something about AWS or Azure GCP, not so much.
It’s not about which company is technically better, but I didn’t want to ignore your technical arguments...
Native integration with G-Suite as an identity provider. Unified permissions modeling from the IDP, to work apps like email/Drive, to cloud resources, all the way into Kubernetes IAM.
You can also do this with AWS - use a third party identity provider and map them to native IAM user and roles.
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_cr...
Cost. Apples to apples, GCP is cheaper than any other cloud platform. Combine that with easy-to-use models like preemptible instances which can reduce costs further; deploying a similar strategy to AWS takes substantially more engineering effort.
The equivalent would be spot instances on AWS.
From what (little) I know about preemptible instances, it seems kind of random when they get reassigned but Google tries to be fair about it. The analagous thing on AWS would be spot instances where you set the amount you want to pay.
Class leading software talent. Google is proven to be on the forefront of new CS research, then pivoting that into products that software companies depend on; you can look all the way back to BigQuery, their AI work, or more recently in Spanner or Kubernetes.
All of the cloud providers have managed Kubernetes.
As far as BigQuery. The equivalent would be Redshift.
https://blog.panoply.io/a-full-comparison-of-redshift-and-bi...
Reliability is just one factor in the equation, and GCP definitely isn't that far behind AWS
Things happen. I never made an argument about reliability.
GCP can be a fair bit cheaper than AWS and Azure for certain workloads. Raw compute/memory is about the same. Storage can make a big difference. GCP persistent SSD costs a bit more than AWS GP2 with much better performance and way cheaper than IO2. Local SSD is also way, way cheaper than I2 instances.
Most folks deploying distributed data stores that need guaranteed performance are using local disk, so this can be a really big deal.
However, I could see doing a multicloud solution where I took advantage of the price difference for one project.
The AWS console is wildly inconsistent. I’ll give you that. But, any projects I am doing are usually defined by a Cloud Formation Template abd I can see all of the related resources by looking at the stack that was run.
Theoretically, you could use the stack price estimator, I haven’t tried it though.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
>We will provide more information by Monday, 2018-11-12 11:00 US/Pacific.
Wait, did the people tasked with fixing this just take the weekend off?
My working assumption is that 18006 should have closed out 18005. But now it sounds like there's a different issue, which we're working to get to the bottom of.
And this is likely a major incident with significant customer impact.
The way google is handling all this gives a pretty poor impression. Seems like this kubernetes is just a PoC.
https://landing.google.com/sre/sre-book/chapters/managing-in...
Looks like this time Mary took the whole week off without telling Josephine :)
Perhaps some of the issues are localized? Perhaps it's even user error (it happens, you know?). But because a small amount of HN users say "it's everywhere!" then suddenly people reach for their pitchforks.
Sometimes we just don't have all the information.
I did this in the australia-southeast1-a zone.
Error message when creating a new Cluster:
Deploy error: Not all instances running in IGM after 35m7.509000994s. Expect 1. Current errors: [ZONE_RESOURCE_POOL_EXHAUSTED]: Instance 'gke-cluster-3-pool-1-41b0abf8-73d7' creation failed: The zone 'projects/url-shortner-218503/zones/us-west2-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later. - ; .
What would a small business do as a contingency plan?
Going multi region on AWS should be safe enough.
If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days.
You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not feel too bad about my small business’ downtime.
Additionally, from a "nobody ever got fired for buying IBM" perspective, you're unlikely to catch much blame from your users for going down when everyone else was down too.
Multi cloud is almost always more pain than gain. You’d spend time and effort abstracting away the value that a cloud provider brings in canned services.
Hell, multi region is often more than many workloads need.
Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage?
There are formal academic ways to determine what your overall availability is, but don't have those on hand. Suffice to say, it takes significant research, planning, execution, and testing to ensure a target availability. (See Netflix https://medium.com/netflix-techblog/the-netflix-simian-army-... ) if someone says they have 99.9% or better up time, they had better have proof in my mind (or a fat SLA violation payout)
People outsource to cloud providers not because they are cheap, but because managing infra in house is hard. Also move fast and break things.
Read AWS docs about availability, there are availability zones in a region, spread across those to minimize impact. Then test when something goes down. Fix/repeat.
Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “price insensitive” enterprises (you can’t just make up a price that’s huge, they do have budgets still) and most of their customers are unwilling / unable to pay more for the extra availability. If your software is designed from the start better, multi region failovers should be fairly inexpensive though. But all the bolted on “multi region” software I’ve seen has been hideously expensive and oftentimes less reliable due to the design being soundly not able to tolerate failures well.
Ultimately it’s a risk/return decision.
“Is going exclusively with AWS/azure/GCP etc a better decision in reliability, financial and mantainability terms than complicating the design to improve resiliency? And will this more complex solution actually improve reliability?”
If AWS ever screws up, you will be able to continue running the business even if it might take weeks to start over.
For live redundancy, you should have a secondary datacenter on another provider, but realistically it's hard to do and most business never achieve that. Instead, just stick with AWS and if there is a problem the strategy is to sip coffee while waiting for them to resolve it. Much better this way than you having to fix it yourself.
Depends on your definition of small. If it's small enough not to have a dedicated infrastructure team designing multicloud solution, then the contingency plan may be: switch DNS to a static site saying "we're down until AWS fixes the issue, check back later".
Otherwise it depends on your specific scenario, your support contracts, and lots of other things. You need to decide what matters, how much the mitigation costs vs downtime, and go from there.
I wish I was only being tongue-in-cheek.
Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services.
This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work
To answer the original question: It looks like this issue was just a UI bug that affected the console, the service itself wasn't impacted. Events that do impact the service will be contained to a region, meaning you can mitigate it with proper redundancy across regions, no zany multi-cloud solution required.
I faced some pretty serious resource allocation issues earlier in the year. The us-west1-a region was oversubscribed. I was unable to get any real information from support with regard to capacity. Eventually my rep gave me some qualitative information that I was able to act on.
One thing I do care about though, is root cause analysis. I love reading a good RCA, it restores my faith in the company and makes me trust them more.
(I'm not affect by the GKE outage so opinions may differ right now!)
Right when I convinced our project to get migrated from AWS...
The timeline of this disruption matches when we started experiencing cloud build errors.
https://status.cloud.google.com/incident/container-engine/18...
Yet, somehow every major cloud provider experiences global outages.
That old AWS S3 outage in us-east-1 was an interesting one; when it went down, many services which rely on S3 also went down, in other regions beside us-east-1 because they were using us-east-1 buckets. I have a feeling this is more common than you'd think; globally-redundant services which rely on some single point of geographical failure for some small part.
We know because we are still waiting here in ap-southeast-2 for services such as EKS to be made available. Pretty sure that any reliance within their backend services on us-east-1 was just a temporary bug and nothing systemic.
Always saying resource not available. My account is a pretty new account.
In contrast, one of my friend is having a pretty old account which is very active. He has no such issue.
So I think due to this issue, Google has enabled some resource limitation for new accounts.
But they should properly communicate this issue.
The specific issue appears to be about creating new "node pools". Creating standard VMs in GCP works fine however, so this is specific to GKE and their internal tooling that integrates with the rest of GCP.
GKE doesn't (at least to my knowledge) allow you to create VMs separately and join them to the cluster in any kind of easy fashion.
An instance in us-central1-a has refused to start since last Thursday or Friday.
I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend.
On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up.
And yet, the status page says all services are available.
Is the typical of others' experiences?
https://stackoverflow.com/questions/53244471/gke-cluster-won...
And for those who have used both, which would you go with today?
Is there another status page Google? Coz the last update I'm looking at...is dated on the 9th..
_If_ that's the case, something else is causing the error messages other people are seeing
I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well.
Monolith for the win! Opinions?
Things like throwing another node into the cluster, or rolling updates are free, which you would otherwise need to develop yourself. All of that is totally doable, of course, but I like being able to lean on tooling that is not custom, when possible.
When your infrastructure does need to become more complicated, you're already ready for it. Even if I were only serving a single language, starting with a K8s stack makes a lot of sense, to me, from a tooling perspective. Yeah normal VMs might be simpler, conceptually, but I don't consider K8s terribly complicated from a user perspective, when you're staying around the lanes they intend you to stay in. Part of this may also be my having worked with pretty poor ops teams in the past, but I think K8s gives you a really good baseline that gives pretty good defaults about a lot of your infrastructure, without a lot of investment on your part.
That said, if you're managing it on a bare metal server, then VMs may be much easier for you. K8s The Hard Way and similar guides go into how that would work, but managing high availability etcd servers and the like is a bit outside my comfort zone. YMMV.
Most monoliths software companies build aren't actually monoliths, conceptually. Let's say you integrate with the Facebook API to pull some user data. Facebook is, within the conceptual model of your application, a service. Hell, you even have to worry "a little bit" about maintaining it; provisioning and rotating API keys, possibly paying for it, keeping up to date on deprecations, writing code to wire it up, worrying about network faults and uptime... That sounds like a service to me; we're three steps short of a true in-house service, as you don't have to worry about writing its code and actually running it, but conceptually its strikingly similar.
Facebook is a bad example here. Let's talk Authentication. Its a natural "first demonolithized service" that many companies will reach to build. Auth0, Okta, etc will sell you a SaaS product, or you can build your own with many freely available libraries. Conceptually they fill the same role in your application.
Let's say you use Postgres. That's pretty much a service in your application. A-ha; that's a cool monolith you've got there, already communicating over a network ain't it. Got a redis cache? Elasticsearch? Nginx proxy? Load balancer? Central logging and monitoring? Uh oh, this isn't really looking like a monolith anymore is it? You wanted it to be a monolith, but you've already got a few networked services. Whoops.
"Service-oriented" isn't first-and-foremost a way of building your application. It's a way of thinking about your architecture. It means things like decoupling, gracefully handling network failures, scaling out instead of up, etc. All of these concepts apply whether you're building a dozen services or you're buying a dozen services.
Monolithic architectures are old news because of this recognition; no one builds monoliths anymore. It's arguable if anyone ever did, truly. We all depend on networked services, many that other people provide. The sooner you think in terms of networked services, the sooner your application will be more reliable and offer a superior experience to customers.
And then, it's a natural step to building some in-house. I am staunchly in the camp of "'monolith' first, with the intention of going into services" because it forces you to start thinking about these big networking problems early. You can't avoid it.
Even if you deploy k8s privately, or over at Amazon, I think there's enough horror stories to make you think twice about the technology.
Then, if it isn't going to be k8s for microservices, what's a more reliable alternative?
The key issue here is that k8s was written with very large goals in mind. That a small business can easily spin it up quickly and run a few microservices or even a monolith + some workers is just coincidental. It is NOT the design goal. And the result of that is that a lot of the tooling and writing around k8s reflects that. A lot of the advice around practices like observability and service meshes comes from people who've worked in the top 1% (or less) of companies in terms of computing complexity. What I'm personally seeing is that this advice is starting to trickle down into the mainstream as gospel. Which strangely makes sense. No one else has the ability to preach with such assurance because not many people in small companies have actually been in the scenarios of the big guns. The only problem is that it's gospel without considering context.
So at what point does k8s make sense? Only when you have answers to the following:
* Getting started is easy, maintaining and keeping up with the going ons is a full time job - Do you have at least 1 engineer at least that you can spare to work on maintaining k8s as their primary job? It doesn't mean full time. But if they have to drop everything else to go work on k8s and investigate strange I/O performance issues, are you ready to allow that?
* The k8s eco system is like the JS framework ecosystem right now - There are no set ways of doing anything. You want to do CICD? Should you use helm charts? Helm charts inherited from a chart folder? Or are you fine using the PATCH API/kubectl patch commands to upgrade deployments. Who's going to maintain the pipeline? Who's going to write the custom code for your github deployments or your brigade scripts or your custom in house tool? Who's going to think about securing this stuff and the UX around it. That's just CICD mind you. We aren't anywhere close to the weeds about deciding if you want to use ingresses vs Load balancers and how you are going to run into service provider limits on certain resources. Are you ready to have at minimum 1 developer working on this stuff and taking time to talk to the team about it?
* Speaking about the team, k8s and Docker in general is a shift in thinking - This might sound surprising but the fact that Jessie Frazelle (y'all should all follow her btw) is occasionally seen reiterating the point that containers are NOT VM's is a decent indicator that people don't understand k8s or Docker at a conceptual level. When you adopt k8s, you are going to pass that complexity to your developers at some point. Either that or your dev ops team takes on that full complexity and that's a fair amount to abstract away from the developers which will likely increase the work load of devops and/or their team size. Are you prepared for either path?
* Oh also, what do your development environments start to look like? This is partly related to microservices but are you dockerizing your applications to work on the local dev environment? Who's responsible for that transition? As much as one tries to resist it, once you are on k8s you'll want to take advantage of it. Someone will build a small thing as a microservice or a worker that the monolith or other services depend on. How are you going to set that up locally? And again, who's going to help the devs accumulate that knowledge while they are busy trying to build the product. (Please don't put your hopes on devs wanting to learn that after hours. That's just cruel).
I can't write everything else I have in mind on this topic. It'd go on for a long long time. But the common theme here is that the choice around adopting k8s is generally put on a table of technical pros and cons. I'd argue that there's a significant hidden cost of human impact as well. Not all these decisions are upfront but it is the pain that you will adopt and have to decide on at some point.
Again, at what point does k8s make sense? Like I said, you ideally should be paining before you start to consider k8s because for nearly every feature of k8s, there is a well documented, well established, well secured parallel that already exists in the myriad of service providers. It's a matter of taking careful stock of how much upfront pain you are trading away for pain that you WILL accumulate later.
PS - If anyone claims that adopting a newer technology is going to make things outright less painful , that's a good sign of immaturity. I've been there and I picture myself smashing my head into a table every now and then when I think of how immature I used to be. Apologies to people I've worked with at past jobs.
PPS - From the k8s site, "Designed on the same principles that allows Google to run billions of containers a week, Kubernetes can scale without increasing your ops team." <-- is the kind of claim that we need to take flamethrowers to. On paper, 1 dev with the kubectl+kops CLI can scale services to run with 1000's of nodes and millions of containers. But realistically, you don't get there without having incurred significantly more complex use cases. So no, nothing scales independently.
Given how both the JS and devops worlds seems to be progressing, is there any reason to believe that this will change before the next thing comes and K8S becomes a ghost town?
Also, migrating to microservices for existing services might not be worth it, especially if you don't operate at a massive scale.
Keep it simple stupid is still a solid design decision, despite all the microservice/container hype.
Most bussinesses only need a couple of servers that provide the service, spread redundantly with a HA capability.
