As a result, we ended up building something that's sort of a "meta-controller programming language" on top of Kubernetes called Koreo. It provides a solution for configuration management and resource orchestration in Kubernetes by basically letting you program controllers. We've been using Koreo for a while now to build internal developer platform capabilities for our commercial product and our clients, and we recently open sourced it to share it with the community.
Koreo has some similarities to configuration languages like KCL, Jsonnet, etc. since it is a means of configuration management (e.g. you can define base configurations, apply overlays, point patches, and so forth). Where it really diverges though is Koreo provides a unified approach to config management and resource orchestration. This means you can start to treat Kubernetes resources as "legos" to build pretty sophisticated workflows. For instance, the output of a resource can be used as the input to another resource. This isn't really possible with Helm, even with `lookup` because `lookup` requires the resource to already be in-cluster in order to reference it.
This is why we refer to Koreo as a meta-controller programming language because it effectively lets you program and compose Kubernetes controllers into cohesive platforms—either built-in controllers (think Deployment or StatefulSet), off-the-shelf ones such as AWS ACK or GCP's Config Connector, or custom operators. It lets you build or combine controllers without actually needing to implement an operator. Through this lens, Koreo is really more akin to Crossplane but without some of the limitations such as Providers and cluster-scoped managed resources.
It seems crazy and maybe it is, but I've found working in Koreo to actually be surprisingly fun since it kind of turns Kubernetes primitives into legos you can easily piece together, reuse, and build some pretty cool automated workflows. You can learn more about the motivation and thinking behind it here: https://theyamlengineer.com
But I guess there is an actual preview implementation now? https://docs.crossplane.io/v2.0-preview/
The comparison on Kro would definitely be good to include as there are quite a few similarities. I can write up more on how it compares in a bit.
One difference is just in how the two approach doing this. Koreo takes an approach of providing primitives that can be composed or reused and, importantly, are actually testable (since testing is a first-class thing in Koreo). This lets you more easily validate automations but also makes it easier to provide "building block" like components that can be shared between Workflows.
Another difference is in how Koreo solves configuration management. Rather than relying on string templating or unstructured YAML overlays, Koreo treats configuration as structured data. This allows you to specify and tweak configurations in a predictable and typesafe way by transforming, validating, and composing them programmatically. Koreo is very much modeled after functional programming principles, so we can, for instance, define functions that validate preconditions or apply standard tags to resources in an environment. This model also enables configuration reuse and overrides across teams and environments without introducing tight coupling or duplication. Instead, we can apply configuration "layers" to build up a resource. Kro really focuses more on resource orchestration and leaves the configuration management up to the user.
Koreo would also allow you to implement a workload spec such as Score rather easily for the same reason. https://score.dev
Looking forward to trying this out soon!
I also think more high level patterns would make Koreo more approachable. Real world problems and how they could be addressed with Koreo.
We're thinking through better examples to try and highlight some of the key differences from other tools without it being too involved.
(Congratulations on the launch, looks interesting!)
We wound up exposing this as YAML purely because there's a lot of tooling out there for dealing with YAML. I am not sure if that's a good reason or not.... Writing the language server was quite nasty because of that choice! Our hope was that the language server and structural type checking (which is very simplistic at the moment) and testing framework can make it feel more like a real language and less YAMLy.
Its toolkits all the way down.
But I’m not sure I feel the advantage of this indirection. It feels confusing to be that the applied resource will be different from what’s in VCS, and the code feels super heavy for what you’re getting. I’ve been at like this and cross plane, and can’t quite grok why this is better than doing it in a classical programming language. But I think I’m wrong, can you help me understand?
Everything is in-code and designed to have a "proper" SDLC lifecycle—code reviews, approvals, merges. It is designed to be used in gitops workflows.
For the more nuanced questions, here's some background: I like Kustomize and KPT a lot. In _my_ opinion, they should be your starting point. They are clean and easy to reason about tools. They do not work as well when you have more complexity. They're very painful if you've got _dynamic_ values or values you need to inject programmatically (think Helm's values.yaml).
The next important item to note: Koreo's relative value is lower if you're building highly bespoke one-offs or you do not care about having standard resource configurations / application architectures. The value is not zero, but there are lighter solutions and you should consider them instead.
Koreo is meant to model application architectures and resource capabilities. Using your example, you can build a BucketResource. That BucketResource will then ensure that S3 Buckets follow your company standards, including things like automatically handling the IAM setup and permissions for the service that uses the bucket. That lets you define required capabilities: An S3 Bucket is always tagged with the owner service and product domain. In production environments buckets must have lifecycle rules specifying a minimum 30 day retention. In development environments, you lifecycle rules are optional. The developers then only need to specify that their workload uses an S3 Bucket and it will be configured based on your company standards. But, we have designed it so that you can decide how much abstraction is right for your needs—you can directly expose the full, underlying API or you can abstract it more.
Effectively, it gives you an "easy" solution for building a PaaS that implements _your_ standards and opinionation.
Our original versions were directly implemented using go and Python. The issue is that iterating on the application models was much, much slower. This approach allows us to rapidly implement new capabilities and features, and even expose unique or experimental architectures to only certain application domains.
There's a comparison against Argo Workflows, but with the description here and in other comments, Koreo seems to be aiming more for what I would use Argo CD for - managing the entire state of the cluster, the controllers, configuration, etc. Because of it tying into repos, you can then define the entire state of your cluster in code, and Argo CD has tools for doing some of the interpolation of variables into your YAML.
The project looks cool, and I don't think that the world suffers from having multiple ways of doing something, I just want to understand it better.
Templating systems are always frustrating, and I couldn't find CLI to spit out exactly what I was going to get. Kustomize ability to build exactly what cluster is going to consume is one of those features you miss when you don't have it.
Tying it to Kubernetes is both good and bad. Alot of companies use Kubernetes so for those companies, this is great. Downside is I think many companies are not ready to deal with complexity of Kubernetes so system that could put them outside of it might be great. That's just taste I guess.
This is a crowded field so good luck I guess.
Finally, the problem here everyone is trying to solve is skill gap and that's hard to fix with technology. Most devs are bad at Ops and that's where friction comes. It's like watching Product Owner develop their feature using LowCode or AI. It works until it doesn't and when it doesn't, here we go. I also realize few companies want us around since they see as pure money sink.
Most of my frustration around building platforms is lack of communication. Most of it due to developers not understanding or just not thinking about it (See skill issue above) so Ops is forced to put in something ugly to get them into Prod at 11 hour so we don't get tossed under the bus.
The CLI is presently being reworked to expose that capability. Right now, you can use FunctionTest in order to validate that you get your expected outputs from an input. This works well because you can test many scenarios, including error conditions. The CLI does not currently emit the materialized manifests purely because our initial use cases needed to map values from reconciled manifests into other manifests. It is completely viable to emit the materialized manifests though.
Honestly, our ambition is to develop a tool that makes the operations and platform engineering people's lives better. Our team is comprised of software engineers who've worked in the operations and platform engineering space for a long time. It is a crowded space for sure, but I am optimistic, probably foolishly so, that we can develop something that is more pleasant for the platform folks than many of the other options.
We'd welcome any other feedback or thoughts on how to accomplish that.
Only in cases where the values are statically known, however. If you have resources that depend on the output of another resource, then we can't know that at "template time" as you pointed out.
What advantages does this offer over rolling my own CRD and operator? Assume it takes me 4 hours to write an operator end to end.
That is how this project started. We wrote custom operators for the various platform components; that worked outstandingly well. Shockingly well to be honest.
This system evolved from that approach because we needed a way to rapidly customize our application models. If you don't need to support varied use cases _and_ have standardization writing your own is the way to go.
Yaml is no go for me.
Gradually typed languages, with support of unkown values, like Nickel can be good.
> However, the two tools are not mutually exclusive. In some cases it can make sense to use Kustomize in combination with Koreo!
Would be awesome Koreo make Kustomize no need. I feel insane getting into project where Helm, Kustomize and Koreo used at same time(they always do this).
So if Koreo and Helm only, would be nice gradual step. One bad yaml with other bad yaml.
In terms of the YAML aspect, internally the core system does not use YAML. We opted to stick with YAML as the interface for the time being purely because it is familiar and there's loads of tooling.
We've been working to improve the structural and the type checking further and I'm hopeful to have a substantial improvement to that released "soon." It is queued up behind some other improvements within the language server we need to make first though.
Most of the current technologies can be virtualized, and with LVM snapshots are a breeze, even extending media it's perfectly done. There's no need to use half-backed namespaces when kernel-level deduplition for memory pages exist when you run similar parallel VM's.
I find virtualisation far easier than containers. Not as fast to deploy, sure; but far more manageable for rollbacks.
