undefined | Better HN

0 pointssolatic1y ago0 comments

If you split feature flags into a different repository, then you're losing the benefit of having everything in a single repository, of having consistency and avoiding situations where your codebase refers to feature flags that don't exist, old feature flags that are no longer checked by the codebase, etc.

At this point, your production infrastructure is no longer solely one stateless server + one database, but two databases: your RDBMS and your GitOps repo tracking feature flags. Do you really get enough value from the second GitOps database compared to putting your feature flags in your main RDBMS?

0 comments

danpalmer1y ago

With one repo you still have versions of this problem – have the flags been deployed as config to the flag system before the server build, has the server build removing usage rolled out fully by the time you remove the config. Using an RDBMS is an option, but makes scaling harder, you still need an audit trail, review processes, etc, so you eventually end up building a source control system on top of it (assuming you hit all the sharp corners and put time into solving them).

If you're feature flagging client code (i.e. somewhere you don't control rollouts, like mobile and web apps) that adds another layer of complexity.

While it's nice to have a simple system, having built one from scratch and used very mature feature flagging systems, my experience is that production systems hit almost all the edge cases quite quickly and flagging/experimentation systems are forced to evolve quite quickly to actually account for these issues.

Multi-repo or not isn't really an issue. My previous company had flag config in a separate repo, my current company has a monorepo, it doesn't really make a difference.

solaticOP1y ago

> Using an RDBMS is an option, but makes scaling harder

Modern Postgres scales vertically quite well on modern hardware.

> you still need an audit trail, review processes, etc,

But you need this anyway for the RDBMS in your architecture. You need an audit trail for when engineers need to get into the production database, and to show that their changes passed review, etc. My point is, if you anyway need to build this for your RDBMS, then you can build on top of that for your feature flag system if you throw that into your RDBMS as well.

> my experience is that production systems hit almost all the edge cases quite quickly and flagging/experimentation systems are forced to evolve quite quickly to actually account for these issues.

I think that's more an argument to use a commercial feature flag platform (like LaunchDarkly) instead of a FOSS option. A commerical platform is anyway what I would prefer to recommend! But, with the context of "choose a FOSS option", it seems to me like building on top of RDBMS, rather than GitOps, makes more sense.

danpalmer1y ago

Many services won't need an RDBMS, and for those that do there's a difference between the control plane of an application (development and releases), and the data plane (users using the service).

This is a complex and nuanced topic, but on my previous team of ~6 where we built a custom solution, we decided against using an RDBMS for multiple reasons, and on my current team where we use the same flagging system across 15 or so >1m requests per second services, there's no way it would work for us. If it works for your use case, that's great! But my advice for anyone else reading would be to put a lot of effort into considering the options as it's hard to change later and has significant impact on how the flagging system is used.

As for whether to use a commercial platform... my preference is probably to build my own with what I need in a system that I can modify as needed, or a commercial platform if there's one ready to go at a good price with the right feature set. I probably wouldn't use an existing open source option here unless I was forking it and treating it as my own from then on, as I find these things need flexibility and customisation. I've yet to see a great open source option.

j / k navigate · click thread line to collapse

0 comments

danpalmer1y ago

If you're feature flagging client code (i.e. somewhere you don't control rollouts, like mobile and web apps) that adds another layer of complexity.

Multi-repo or not isn't really an issue. My previous company had flag config in a separate repo, my current company has a monorepo, it doesn't really make a difference.

solaticOP1y ago

> Using an RDBMS is an option, but makes scaling harder

Modern Postgres scales vertically quite well on modern hardware.

> you still need an audit trail, review processes, etc,

> my experience is that production systems hit almost all the edge cases quite quickly and flagging/experimentation systems are forced to evolve quite quickly to actually account for these issues.

danpalmer1y ago

Many services won't need an RDBMS, and for those that do there's a difference between the control plane of an application (development and releases), and the data plane (users using the service).

j / k navigate · click thread line to collapse