Specifically in our rules everything after the "if" is Common Expression Language.
See: https://firebase.google.com/docs/firestore/security/rules-co...
The efficiency and safety of CEL enables us to put security rules in the critical path of every database request.
This really needs to be stated on its front page. Right now it’s all “Hows” and no “Why”.
First question anyone looking at it asks: What problem does it solve?/What need does it fill? A real-world use case provides an easy relatable answer.
Incidentally, with existing links to protobuf and no halting problem to worry about, it sounds like you’re halfway to having a remote query language a-la SQL too.
https://www.researchgate.net/publication/221553413_Safe_Quer...
Google seem to like these executable config languages because they've got another open source one ("Starlark") a few notches up in expressivity.
1. Build an AST on top of YAML a la CloudFormation. Now you're programming in YAML, hurray!
2. Extend your static language with executable features a la Terraform/HCL, basically reinventing (and very badly, at that) more traditional language features
3. Use text templates, a la Helm--now you can generate syntactically invalid configuration! (and absolutely trivially, at that)
4. Use an expression language (familiar, ergonomics) a la Pulumi, Starlark, Nix, Nickel, Dhall, etc
Note that in these conversations, someone inevitably shouts "use the simplest tool for the job!" ignoring that static configuration languages (and options 1-3 above) are strictly more complex for the reusability use cases outlined above.
EDIT: Pulumi isn't an expression language; rather, it lets you use real languages to generate configuration, and these languages often include powerful expression features. AWS's CDK is also in this category.
I think one problem is that configuration needs start out simple and evolve to complex - as opposed to being obvious from the start you'll need dozens of discrete components to deploy. At that early stage, anything beyond a few lines of YAML seems like definite overkill.
Eventually, it becomes clear that it's very hard to maintain, but by then there's thousands of lines of hard-knocks battle-tested, working production config to try and basically recreate from scratch, without breaking anything.
Until you've gone through this process once (or maybe a couple times..) it's hard to see why you should make that initial leap to a much more complex, tools-required workflow. "But eventually, we will probably need..." is a tough sell against someone arguing to do the simplest thing.
Me too. I feel like there should be some eponymous law about this. Every declarative language that starts out trumpeting "simplicity and not being Turing complete is a feature!" ultimately grows features until it is an imperative language, or gets replaced by one that is.
If you're gonna get there anyway, you may as well design for that instead of bolting on features poorly after the fact.
Nearly every programming language has a YAML serialization library†. And before that serialization happens, your config can be expressed using the regular-ass coding features of your program, however you like. (For optimal clarity, I personally would suggest creating a builder DSL and using it.)
† Technically a language doesn't even need a YAML serialization library to emit valid YAML; because valid JSON is also valid YAML. You can just serialize to JSON on your end, and feed the result into anything that's expecting YAML.
I agree, but it becomes very important to limit scope. For example, Azure templates allow looping and conditionals and all sorts of fancy stuff. Approaching a typical library of ARM templates is a massive undertaking: reading a JSON (or YAML) if statement is not ergonomic in any way - causal relationships can be multiple screens (or files) apart because of the sheer amount of JSON required to represent executable code.
It should be kept relatively lightweight, with stuff like CloudFormation GetAtt to glue deployed things together. Anything more complex should be solved with tooling designed for computation, i.e. programming languages (that emit config, e.g. Pulumi).
The moment you encode alternation in the configuration file, its time to think about biting the whole Turing complete bullet.
- file:
path: /etc/foo.conf
owner: foo
group: foo
mode: 0644"CEL evaluates in linear time, is mutation free, and not Turing-complete. This limitation is a feature of the language design, which allows the implementation to evaluate orders of magnitude faster than equivalently sandboxed JavaScript."
As mentioned, the goals are security policies (it was first used internally as the Security Rules for Cloud Storage for Firebase and the Cloud Firestore) and proto contracts (e.g. you could define addons to your proto to specify the data matched certain behavior):
I forget the exact syntax for the contract, but it looked something like this...
``` message person { @contract(matches(/* RE2 phone number regex */)) string phone_number = 1; ... } ```
That data could enforce client side checks as well as be used server side (in different implementation languages).
I always wanted to see it combined with the proto to Firebase Security Rules generator (https://firebaseopensource.com/projects/firebaseextended/pro...) to do client and server validation.
Sort of. Starlark doesn't (or at least didn't originally) support recursion or while loops or a number of other structures. There's also a few other differences that make starlark "better" for configs (some immutability is different, there's no such thing as a `class`, etc.)
I still support loops in a configuration language
for x in sequence:
generate_complex_thing(x)
[generate_complex_thing(x) for x in seq]
Cartesian product, map, and reduce operations over finite sets and lists are really handy in configuration. ("For each server in SetA look at each path in SetB and ...") But, if you find yourself starting to write general loops (as opposed to loops implementing map, reduce, and Cartesian product in languages that don't have them built-in), it's a sign you're starting to blur the line between configuration and business logic.
Unit-testing configurations is difficult, especially if they can be non-deterministic (depend on data/time/random()) and aren't modular.
In some sense, all programs with configuration files are really interpreters for the language of their configuration files. (As mentioned before, many of these abstract machines are just push down automata.) Taken too far, the configuration becomes the real program.
I've seen a (now retired) automated trading system with a powerful XML-based configuration language where a few times people got themselves into trouble (and caused trading losses) when their complex tower of configuration fell over. Part of the problem was there existed a few people who weren't trusted to write application logic, but who were trusted to "just update configurations". When the only tool some of your people are allowed to use is a hammer, hammer marks start mysteriously showing up everywhere. Additionally, this was over 10 years ago, and prior to these trading losses, configuration underwent less stringent review. I don't think my experience was atypical.
I've also seen configuration loading get stuck because someone added some code to the config to hit a REST endpoint in the middle of the configuration file. Ideally, you'd leave any I/O to the main program logic, where it's easier to perform the I/O asynchronously, or otherwise non-blocking.
Deterministic non-Turing-complete immutable "executable" configuration languages (or at least ones where it's difficult to get unbounded recursion) tend to be a happy medium. Also, declarative rather than imperative configuration languages tend to be easier to read.
Back when I was a developer in web search infra at Google, I vaguely remember once or twice using a language (maybe Borg's config language, borgconfig) that completely lacked mutability and essentially used object prototyping (A is created as a copy of B, with differences specified at object creation time.)
This ... yes. Being the ops guy backing up second line support at an ISP for a while brought me many examples of this and inspired many in-house tools for them to use instead.
- hard to write exploits, can be shared/installed without warnings
- easy(-er) to predict behavior of themes so they won't break in new versions
> Now, I never intended for the file format to become a scripting language—after all, my original view of Ant was that there was a declaration of some properties that described the project and that the tasks written in Java performed all the logic. The current maintainers of Ant generally share the same feelings. But when I fused XML and task reflection in Ant, I put together something that is 70-80% of a scripting environment. I just didn't recognize it at the time. To deny that people will use it as a scripting language is equivalent to asking them to pretend that sugar isn't sweet.
https://web.archive.org/web/20041217023752///x180.net/Journa...
Initially it was designed to process incoming slack messages, and sometimes trigger a notification to an on-call engineer, but over time I've found uses for it processing email, scripting simple actions on my desktop, and more.
https://github.com/skx/evalfilter/
These kind of things are pretty simple to write, but sometimes I almost think it is a shame there isn't something more standard. (Lua was kinda winning for that embedded-logic role for a long time, but nowadays we still have the mixture of YAML, HCL, and other niche-specific language/filtering and I imagine the time has passed to pick one standard.)
I believe this was a mistake. "?" based conditionals aren't really a good idea IMHO.
Instead of `<cond> ? <left> : <right>` I prefer `if <cond> { <left> } else { <right> }` the additional brackets noticeable improve readability and you can extend it to support `if <cond> { <a> } else if <cond2> { <b> } else { <c>` instead of `<cond> ? <a> : <cond2> ? <b> : <c>`. (Oh and that last example might be wrong needing brackets depending on operator precedence...)
Through if you don't nest it it doesn't matter (oh and because it's a expression evaluation `else` is not optional but required as you need a value the expression resolves to).
If you want ABAC, use OPA. In every case.
CEL, and CEL-GO, is entirely different. It allows you to evaluate arbitrary expressions with random data. Think a search (eg. linkedin API's crappy search, or log searching, or random predicates).
You would not define complex policies in CEL like you would in OPA. Well, I would not - you can define arbitrary macros and functions in CEL but it is not made for that scale. OPA is more suited for that.
Some examples:
- In OPA, you can define a policy that matches RBAC, ownership/acl, and ABAC in one file. With multi-tenancy. Think: "as a patient, I can see my data", and "as the patient's guardian, if they're under 18, I can see their data". And "As a doctor in the patient's clinic, I can see their data". And "as a clinical director in sudo mode, I can see their data". All in the same policy package, with tests.
- OPA supports "partial evaluation". For example, if you only have a subset of data available, you can evaluate an OPA policy and have OPA tell you whether the policy evaluates to true or what data is missing. This is quite powerful for building up complex auth layers.
- In CEL, you can say "all users > 30 days old". Simple, easy, filtering. EG, with a custom date macro, `date(users.created_at) > duration("30d")`.
In short, use both. OPA for security and complex policies. CEL for user-defined "expressions".
My biggest problem with a lot of these generic computation (you could view OPA as generic computation but with a focus on auth) is that they bring their own DSLs -- I'd love to see something like CEL that's based on regular programming languages, and the only way I can think of doing that right now is through WASM.
I left the team about that time, so I don't know what exactly happened after that, but I wouldn't be surprised if the two are fairly close. My assumption is that's why CEL is polished up and OSS (I think we first published it a few years ago, why'd it get posted now?)
https://github.com/google/cel-spec/blob/master/doc/langdef.m...
A lot of the times engineers at Google will open source libraries or tools they have worked on, which go under the Google GitHub repo, but are attached with that language. This is basically saying that it is owned by Google but it is not something Google is officially supporting. It may continue to get updates, it may not. I've definitely seen some libraries open sourced from Google, that stopped being pushed externally once the primary driver behind it left Google or moved on to other projects.
You can actually read some of the process that a Googler will follow when open sourcing software here: https://opensource.google/docs/releasing/
Too bad it isn’t open sourced.
And then you have an easy mechanism to allow some configs from trusted parties to be a bit more capable, if they need it.
But now I see, since it has recursion, there are tricks to make it go in an infinite loop.
Thanks for the pointer. I read the rationales again in the doc.
Fast - CEL runs without the need for sandboxing, making it much faster than sandboxed solutions like WebAssembly, Lua, and embedded JavaScript.
Scalable - Features like variables and functions would make CEL more expressive, but also less scalable as it's easy to write a few lines of code with functions that consume exponential amounts of memory and compute. CEL is simply the expression and nothing more.
Portable - CEL is implemented in Go[0], C++[1], and Python[2] with Java open sourcing in development. There is a public codelab[3] available for Go if anyone is interested. There is also a conformance suite in CEL-Spec to ensure consistent behavior between runtimes and environments. Our objective is to make it possible to bring CEL to K8s, J2EE apps, and C++ proxies. Evaluate at line-rate everywhere. Personally, I hope someone tries to make CEL work on IoT devices some day too.
Where? - CEL is usually embedded into larger projects rather than being the one stop shop for solving a particular kind of problem. For example, CEL Policy Templates[4] has an opinionated way of using CEL to validate/evaluate YAML configs. Most of the time CEL is part of a service API.
In addition to being used in Firebase's Cloud Firestore / Cloud Storage security rules, it is also used in several other Google Cloud services: - Cloud Armor[5] - IAM Conditions[6] - Cloud Healthcare Consents[7] - Cloud Build Notifiers[8] - Security Token Service[9] - Access Levels[10], and more.
CEL is also used in some prominent open source projects like Envoy RBAC[11], Caddyserver[12], Krakend.io[13], and Cloud Custodian[14].
[0]: https://github.com/google/cel-go [1]: https://github.com/google/cel-cpp [2]: https://github.com/cloud-custodian/cel-python [3]: https://codelabs.developers.google.com/codelabs/cel-go [4]: https://github.com/google/cel-policy-templates-go [5]: https://cloud.google.com/armor/docs/rules-language-reference [6]: https://cloud.google.com/iam/docs/conditions-overview [7]: https://cloud.google.com/healthcare/docs/concepts/consent-mo... [8]: https://cloud.google.com/cloud-build/docs/filter-build-notif... [9]: https://cloud.google.com/iam/docs/workload-identity-federati... [10]: https://cloud.google.com/access-context-manager/docs/custom-... [11]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... [12]: https://caddyserver.com/docs/caddyfile/matchers#expression [13]: https://www.krakend.io/docs/endpoints/common-expression-lang... [14]: https://github.com/cloud-custodian/cel-python
I am also interested to contribute.
Don't expect your config files to terminate when they use macros, that's all I'm saying.
It's still useful for a config language because it makes it harder to accidentally make a config that (in practice) never terminates, and usually allows for easier static analysis and refactoring of the config files through immutability and purity.
Also it means you need to be careful about malicious input, you need to take countermeasures when you evaluate an expression from an untrusted source.
How is this compatible with their claim that "CEL evaluates in linear time"?
Nevertheless, there's two answers I can think of here. The first is simple (but IMO probably not right): 3-SAT is (probably) exponential, but the language definition document qualifies the linear claim by saying that macro expansion can also be exponential.
The second is probably more accurate, though. General 3-SAT is (probably) exponential, but we're dealing with 3-SAT for a constant number of variables. You can solve 3-SAT by producing all combinations of values for the variables and testing the expression for each one, but while testing an expression should be linear there's an exponential number of combinations. If your combination count is constant, though, the whole complexity becomes linear ... with a shockingly bad constant factor.