Full disclosure: made by a former LaunchDarkly employee + current fan.
Open to missing something though, curious what others experience has been
The distinction is that you have a different release process, or build a different artifact, from your main codebase. The codebase you are controlling with flags doesn't change when your flags do. This can be done with separate repos if you want one build per repo, but it doesn't have to be.
At this point, your production infrastructure is no longer solely one stateless server + one database, but two databases: your RDBMS and your GitOps repo tracking feature flags. Do you really get enough value from the second GitOps database compared to putting your feature flags in your main RDBMS?
Flag changes can be pushed directly to the main branch with the correct repo permissions. When using the GitHub UI this involves just a little bit of typing and a few clicks.
>might as well just change the code at that point
If changing the code, running tests, building, and deploying is quicker and less risky then yes that makes more sense.
The best implementation I've seen was in a Java project. Features where enable or disabled by either the properties file or the database. If a flag was set in the database, then that took precedence. New features would always be rolled out disabled in the properties file. Then in a controlled window the new features would be enabled for a few minutes and logs would be examined. If everything looked good the feature would then be enabled again. After a few days or weeks, the properties file would be updated to have the feature enabled by default and the flag in the database deleted in a later task.
> I'm about to do this thing to mitigate the issue, does it look like the right thing?
It doesn't need to be a code change, can just be a flags change, but if it's a change at all then why not pin it to a commit so that rolling it back is easy and so that the commit sha can be an indicator of which flags are where.
But Git-style version control with history, diffs, branches and pull requests are pretty useful for feature flags and other "app configuration".
Version history and diffs are great for knowing what flag logic changed when + debugging what broke prod.
Branches let you test and preview flag logic changes in your own isolated branch (which you can point the SDK at) — this is a cleaner approach to having a few separate "environments" like development, staging, production which can drift from each other.
Branches are also great for refactoring the schema / structure of all your flags, e.g. deleting a bunch of flags in one go.
Pull requests and approvals are great for when you're making changes to sensitive flags. E.g. you can lock down specific flags.
Pull requests are also great for onboarding nontechnical team members like PMs or sales reps so they can safely make flag changes themselves but require approval from an engineer (at least while they learn to use the system). Empowering nontechnical people is also why a UI is important.
Branching and pull requests are also a great way to prevent conflicts / overwriting other team members flag changes.
So Git-style features are pretty useful, but you also want the UI and you only want to enforce pull requests for specific flags or team members — this is what we built at Hypertune.
I’m not sure I have a comprehensive solution. I just know which ones I hate more than which others. The repo is the least obnoxious of the options.
For systems of sufficient scale, it's fairly standard to keep flag changes outside of git so that they can be flipped without a pr. That way the flag change UI can apply other validation steps before any change is attempted such as ensuring valid enrolment ranges (no accidental overlap, and no accidental rollouts to 100% instead of 10%), and the associated rollout analytics can be shown alongside the changes.
You can also override things in emergencies more easily, which is the parent's point.
In particular, a change to the yaml for feature flags could bypass most of your build and test pipeline, and changes could be deployed more quickly.
OTOH, you still need to figure out a deployment strategy for it.
1. The time to delivery is potentially much much shorter.
2. There's a built-in rules engine for targeting. You could integrate this in! But it feels nice having it separate.
As you add more of them though, they add tech debt and make the code harder to reason about. Developers are rarely motivated to clean them up after rollout.
In Limited Work In Progress (LWIP aka WIP-Limited) processes, if you are blocked on your story you help someone else get unblocked on theirs. And then escalate if there is no movement.
Been doing that with my new hires in other jobs.
At the end of the day it’s still fuckery I was unable to avoid.
The best companies I've been at where those where product and engineering really operate as one team.
It's stakeholders not giving engineers necessary time for upkeep, probably caused by engineers lacking ownership, or failing to communicate
We already run a giant pool of relays for our apps to connect to, both to save ingress/egress costs, and to provide a backup if LD blips out for a while.
[Edit: yep, it actually uses the Relay Proxy, in offline mode. See https://github.com/dorklyorg/dorkly/wiki/5.-Architecture]
In fact, their entire API for flag manipulation and definition (essentially all the operations you can do with their UI, but programmatically) is open too: https://app.launchdarkly.com/api/v2/openapi.json
One could imagine running an OpenAPI → Go/Java/Rust/Python/etc. code generator on their OpenAPI spec, and then gradually implementing their entire API as open source.
Not sure how they would feel about that… although I suspect more usage of an LD-compatible API would only be good for them.
Of course, it makes sense for all those things to be open. The majority of smaller startups they're competing against will have settled on OpenFeature. DataDog too releases some equivalently surprising things as open source: eg. https://vector.dev/
If this is a limitation for anyone I'm happy to work with you on modifying the terraform module to support HA.
Which yaml files is one supposed to edit to actually modify the flags?
The flags are really useful for things like enabling just a fraction of traffic, or ensuring you can switch a feature off much quicker than a full deploy would take.
In most such cases you have several instances of your backend running in parallel for scaling and redundancy and when making a release, instances of several versions run concurrently. So you don't have a "atomic upgrade" available
With multiple services coordinating upgrade is even harder.
Patterns like expand-contract helps you manage this..E.g. first add the new endpoint to server, then move clients over, then remove old endpoint.
So..feature flags is just a way of dragging this process over longer time period, and roll over % of traffic. Instead of coupling changes to service releases you roll over using config.
Used them a lot in backend to backend, backend to DB etc scenarios, has been hugely useful to us and would never work without it.
But, of course depend on context what you are doing.
be happy to walk you through how we use it. Shoot me an email if you want to chat. wayne at inaccord.com
We almost never shared flags across our fairly chunky servicees. We usually found a softer way to do it.
Even with Consul, you can still have enough skew that a few requests in the middle might see A and !A if your request rate is high enough. So it depends on your business model and architecture if that’s acceptable.
... though there are only three officially-supported OpenFeature adapters so far (Java, NodeJS, .NET)
Probably answering my own question, but the main reason I can think of would be if your app doesn't have some kind of business admin panel capability.
I suppose my bias is also that in my sphere of web dev, we build these business panels pretty frequently, so feature flags and a UI for toggling them is something that makes sense to do within the app rather than add a third party service (and associated cost + potential latency) for it.
Thankfully GitHub has an easy web-based Pull Request process. It's not as simple as a custom UI, but could be used by non-engineers to change flags.
That's a licensing thing, not a feature flag.
Unaudited, untracked web UI based feature flagging has been my peeve with the other solutions. I'm surprised all the feature flagging solutions out there don't have this as a default. The next step would be to use an artifact versioning repository to decouple from S3.
Checking now they have about 4 million subscribers. Probably not gonna be great for SEO longterm.
also can use GitHub
https://github.com/theogravity/feature-manager-wrapper
Edit: It looks like it's a backend replacement to LaunchDarkly, but you can still use the LD client from what I'm reading here, so there's nothing for me to integrate here.
