With respect to the choice of helm: we started out rolling our own pipeline with sed and awk and the usual suspects. When that became too complex helm was just taking off and we moved to that. We still use it to install various charts from stable that implement infrastructure things. For our own applications we found that there was just too much cognitive dissonance between the helm charts and the resulting resources.
Essentially the charts and the values we plugged into them became a second API to kubernetes, obfuscating the actual API below. The conventions around the "package manager" role that the tool has taken for itself also contribute to lessen readability due to scads of boilerplate and name mangling. We recently started deploying things in a new pipeline based on kustomize. We keep base yaml resources in the application repo and apply patches from a config repo to finalize them for a given environment. So far it's working out quite well and the applications engineers like it much better. Now with kubectl 1.14 kustomize's features have been pulled in to that tool, something I have mixed feelings about, but at least the more declarative approach does seem to be the way the wind is blowing.
kustomize came along just as it’s become untenable for us to copy/paste config to multiple environments. I like that it’s pretty much the simplest possible way to customize yaml, and plan to dive in soon.
Kustomize is great for getting a job done quickly if you don't mind some duplication of code and effort throughout your projects, but Helm is ideal for managing dependencies and templating across a large amount of projects where you may want to re-utilize other charts.
I think it is ideal to not just discover these tools by reading about them, but actually start using them and spend a few hours trying them out and testing what they can and can't do.
> running in your cluster that’s also trying to reconcile desired state with actual state
is not needed anymore
Helm templating is pretty terrible. Whoever thought generating YAML as text was a good idea deserves a solid wedgie. But it gets us where we need to be. During our prototyping of our GKE environment, we had lots of individual YAML files, which was not tenable.
Atomic deploys/rollbacks is essential. What Helm brings to the table is a high-level way of tying multiple resources together into a group, allowing you to both identify everything that belongs together, and to then atomically apply the next version (which will delete anything that's not supposed to be there anymore). Labels would be sufficient to track that, in principle, but you still need a tool to ensure that the label schema is enforced.
We don't use any of the other features of Helm -- they're just in the way. We don't use the package repo; we keep the chart for every app in the app's Git repo, so that it's versioned along with the code. We've written a nice wrapper around Helm so people just do "tool app deploy myapp -e staging", and it knows where to look for the chart, the values by environment etc. and invoke the right commands. (It also does nice things like check the CI status, lint the Kubernetes resources for errors, show a diff of what commits this will deploy, etc.)
I've looked at Kustomize, and I don't think it's sufficient. For one, as far as I can see, it's not atomic.
I'm hoping a clear winner will emerge soon, but nothing stands out. My favourite so far is Kubecfg, which is similar to the unnecessarily complex Ksonnet project, which has apparently been abandoned. Kubecfg is a very simple wrapper that only does Jsonnet templating for you.
I'd be interested in how Google does these things with Borg. My suspicion is that they're using BCL (which Jsonnet is based on, last I checked) to describe their resources.
Until now I've used Jinja2 templates for our Kubernetes definitions with a variables file for each environment, but this is awfully manual.
I'd love Kustomize to be sufficient for us as it's poised to become a standard thanks to now being part kubectl.
Unfortunately, in some ways its YAML patching philosophy is too limited, and coming from a templating system would be a step back even for relatively simple use cases : for example, you're very likely to need a few variables defined once and reused across k8s definitions (a host or domain name, project ID, etc). You can't really do that in a DRY way with Kustomize.
AFAIK, it also currently doesn't have a good story for managing special resources like encrypted secrets : it used to be able to run arbitrary helper tools for handling custom types (I use Sealed Secrets), but this has been removed recently for security reasons, prior to the Kubectl merge.
Kapitan seems to cover these grounds, and it doesn't carry the weight of those Helm features which are useless for releasing internal software, but I'm still a bit worried about the complexity and learning curve for dev teams.
Is there anything else out there that goes a little further than Kustomize, is simpler than Kapitan and Helm and fits well into a GitOps workflow ?
Kustomize just applies structured edits to yaml. We run it to apply all the patches and output a single manifest file with all the resources, then send that to the master with kubectl apply. I suspect its as atomic as anything helm does, but I could be wrong.
Sure, it'll manage sets, but will it really flip versions in an atmoic way and does this really matter when it's doing 3 rolling upgrades, without anything to manage which traffic goes where?
It's possible it does more than I think it does, but I'm also wondering if atomic is the right word here?
Yes. Kubecfg is the closest equivalent for k8s. And it also works the best for me (but I might be biased).
Which does the grouping-of-resources thing and explicitly leaves templating up to you.
> I’m curious about looking into kustomize, but the package management/rollback capabilities of helm are quite nice; is there a good “pre-baked” solution for that that doesn’t involve helm?
Not that I'm aware of. I'm only really familiar with helm, kustomize and ksonnet at this point, and of the three only helm took the approach of running a server-side stateful thing that could take responsibility for pseudo-resources called releases. I haven't followed work on the next version closely, but it will be interesting to see what changes as they ditch the client-server architecture. I assume it will be more like terraform with some sort of state file.
Rollbacks for us basically mean go back and redeploy the last good ref. The pipeline is completely deterministic from the commit. The underlying build system that produced the container might not be, but we don't have to rebuild that since it's still sitting in the registry.
Helm seems a weird choice for B.
It's nothing too surprising :). Simply a preference for simple, compose-able tools. I personally feel like patching yaml client-side is outside the scope of a "ctl" tool.
https://cloud.google.com/solutions/migrating-ruby-on-rails-a...
Disclaimer: I work for GCP and wrote most of that :D
At the end of the day, Heroku and GKE are rather different beasts with different philosophies, so migrations are never going to be 1:1. I expect this to become simpler over time as tooling matures though, eg. using https://buildpacks.io to build Docker images instead of having to craft them by hand seems promising.
On a related note, if you have tried it, what do you think of their Cloud SQL MySQL offering?
As far as I know, MySQL 5.7 is as far as they go and like PostgreSQL, they don't support point-in-time recovery. Also, perplexingly (at least as of a year ago?), deleting the instance deletes all backups associated with it, so there's an opportunity to accidentally blow away all your data. I'm sure Google can recover it, but you'll have to submit a support ticket for that.
We don’t use MySQL.
One time the Postgresql instance got stuck during a migration. There was no way of creating a fresh instance with a recent snapshot. Luckily Google's support team managed to unstuck the instance but that meant 24h of downtime.
This part seems very hand wavy, given that Heroku Shield would've solved many (all?) of their problems.
> We were also running into limitations with Heroku on the compute side: some of our newer automation-based features involve running a large number of short-lived batch jobs, which doesn’t work well on Heroku (due to the relatively high cost of computing resources).
How much memory did their batch jobs actually need? If they're using Rails, then I'm assuming they're just running a bunch of Sidekiq jobs that are querying PG. I'm surprised that they'd need that much in terms of compute resources. They should be able to get very, very far by making PG do a lot of the work, or by streaming data from PG and not holding a lot of data in memory.
Even if they did need all this, the following two options seem WAY easier to manage:
1) Use dokku to run your super-intense Sidekiq batch jobs on beefy EC2 instances. You can still schedule them in your Rails app in Heroku, no big deal. Many engineering teams have to do this type of split-up anyway when it comes to Application Engineers and Data Engineers, this is just a simpler way to do it.
2) Similar to 1), use a different language runtime for the batch jobs. If you really need to run CPU intensive jobs, why are you using Ruby? If the jobs aren't so intense to mandate maintaining two languages (fwiw, not that hard), why will moving to k8s solve the issue?
Personally, I'm not sold on their decision to move to Kubernetes, and I use Kubernetes for my job.
Author here; I don’t want to go into too much detail, but we tried Shield early on and had a negative experience that made us wary about using the platform (it seems to use a different tech stack under the hood from “normal” Heroku and lacks a lot of the things that make Heroku great). Also it’s very expensive compared to VPC-based solutions on AWS and GCP.
W.R.T. the batch jobs, I think I didn’t explain super well—we are using a different language and runtime from our “normal” background processing jobs (which use worker queues in Rails), it’s just that Heroku isn’t very well suited for the use case (which is basically FaaS-like but with long-lived jobs).
The “split” workflow you described is basically what we were doing (but with AWS Batch instead of Dokku); it’s just that it’s more cost-efficient to consolidate everything into one cluster (especially with preemptible gke nodes) and also better to have a common set of tooling for the Ops team.
To be fair, we haven’t yet completed the move from Batch to k8s so it’s possible that part of the plan won’t pan out as expected.
Heroku Shield is a service added on top of Heroku Private spaces.
You usually don’t need Shield unless you want to be compliant with things like HIPAA, etc
Which of course could be your case here.
We were about to use Heroku Shield at a previous gig. It's definitely expensive, but at our requirements it was still less than an engineer. I wouldn't run a ton of "big data" processing on Heroku nodes though. I'm sure/hope it exists, but I haven't seen a Heroku-ized version of data processing.
I've never tried, but I've heard a lot of people saying it's very hard, but people are often complaining about the most basic stuff as being hard.. Sooo..?
I maintain two prod and two test environments: AWS+kops (since before EKS existed), Alibaba+kubeadm (because China), and two local ones that used to be minikube but are now just kubeadm. I spent a total of about two months getting the knowledge to do that, and these days I spend about 4 hours a week on maintenance. I do minor version upgrades but put off major ones because there’s not much business value and upgrades can break stuff.
It’s my least favorite part of my job. I’d rather be coding or doing architecture, which is what I spend most of my time doing. We’ll be moving the AWS stuff to GKE in a couple months. Still haven’t heard much on the quality of Alibaba’s offering. Their IaaS is solid but some of their other services will sometimes throw errors that return 0 hits on google, no documentation, which scares me.
AppEngine has an enormous number of limitations that you only hit once you scale up and gets very expensive very quickly.
This is a huge success story that I was part of: https://cloud.google.com/customers/gearlaunch/
At our peak, we were processing $500,000 of online orders per day and spending about $5k/month on our entire GPC bill. It was a rounding error compared with our income.
The biggest expenditure was actually on the Postgres database servers that we needed for our analytics, not on AppEngine.
I have used AppEngine in production for about 50+ clients and I am genuinely really curious to know what these limitations are. Maybe it is dependent on the programming language/framework? I run Phoenix/Elixir with Vue.JS + PostgreSQL as standard for most of my clients, it's really a breeze to work with.
In addition, these are the advantages of working with AppEngine:
Bonus points if it explains what containers are and differences between EKS, ECS and others.
the tech details are irrelevant in posts like this. no one, but no one, is throwing down knowledge and insight that anyone and their mama doesn’t know or can’t easily acquire. people just don’t give away their secret sauce that easily.
the 2 points are
a. a recruiting signal. it lets candidates know “we are like you”. we care about the tech stack and what we do. we care about elegant and “good” solutions.
b. it gives company devs a public sounding board. a chance to have a bigger voice than in house obscurity.
if you think these blogs are really about the tech, well then yes they are quite boring.
Unless current provider has horrible support/issues which you want to specify to help others, these articles are useless.
HN community making these articles to front page which neither describes architect challenges/scalability challenges is pointless
Or you could hire few cloud infra devs and stick to autoscaled VMs (EC2), not spending time on k8s? I bet k8s took you a while to set up and maintain.
