The Current Secrets Rotation Process Is Broken (opens in new tab)

So much practice around secrets and identity is oriented around assuming secrets are expensive. Like, people will set up a database and create ‘the database user’, with its one password which now needs to be used by all the systems that access that database.

To maximize security you need to treat secrets as being cheap. In fact they need to be disposable.

In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password - and, bonus, only the permissions that that particular client needs.

If that secret gets compromised, no biggie - you can destroy that user account and the credential will never work again.

Secret rotation is a related approach, but again - if your rotation frequency is ‘quarterly’ rather than ‘hourly’, is there a good reason for that, or have you again just assumed secrets are expensive?

>And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment

Oh just today I found out the previous team stored all secrets in git in multiple repositories, and now I need to rotate them all. They encrypted them eventually with sops but did not rotate them so you can find everything in git history. The only solution I can come up with so far is create new credentials and gradually migrate everyone to new secrets, and then delete the old ones. And there's still a chance we can forget something and something will break. I wish there was a button "rotate secrets"...

I find it frustrating that people still write programs without thinking about secrets rotation. It's kind of understandable that older stuff didn't think about it, since there was certainly a time when people treated secrets, especially things like API keys, as effectively indefinite, but that just doesn't make sense, especially not now. Same for any ephemeral keymatter, or other cryptographic keymatter: there should be a way to have multiple, with at least one "primary" secret and at least one secondary secret. The way I often do this is as simple as having an environment variable/flag/whatever for two separate values and try them in order. Rotating secrets used to salt or encrypt database entries is definitely a bigger pain in the ass, but if it's a big pain in the ass to design a system for it when you're building out a feature, you'd better bet it will be a pain in the ass when you've got a huge security breach and are faced with the task to do it not only in short order, but in concert with many other tasks related to the ongoing fire.

There are definitely other things to be said about the state of secret tokens and secret management, but this one is honestly not that hard to do significantly better at. Rotating secrets should not involve downtime, it should be something you can do out of precaution...

You're exactly right about those two problems, and they actually go together quite well. If a user who previously had access to a secret suddenly has that access revoked, you should rotate that secret. Technically that's only really necessary if the user ever actually saw that secret value, but most secret managers don't expose that information.

At Doppler, we're currently in the process of building out our rotation engine[0]. It allows you to automatically rotate secrets with third party providers (including your database) on a schedule you define. We also provide access logs to see exactly who saw what and when. The ultimate goal is to automatically rotate your secrets as users' access changes. And if you get notification of a breach, you can rotate everything with a single click.

[0] https://docs.doppler.com/docs/secrets-rotation

The last part is the key thing. I'm in a continuous running battle with our information security risk team over this - they believe that secret rotation is a secrets management platform feature (and we use Vault so that box is ticked) but the reality is that secret rotation that isn't tied to restarts / reloads of said secrets within distributed application estates is just another way to have an incident.

This is particularly true if you've backed into secrets management as a practice and retrofitted existing platforms; but even if you've designed net new, it's an abstraction-breaker to get reloads of credentials happening all the way down the stack.

Even a blue/green deployment isn't a panacea if there're databases or other shared states that depend on the credential cutover.

From my point of view, the better approach is not to share secrets at all where possible and drive everything from service identity directly. e.g. short-lived credentials like AWS IAM or short-lived PKI proofs like SPIFFE.

I'm having the same success with Google IAM, and I agree it completely solves the problem, except for one thing; It only works with compatible services, within your company boundary, within the same cloud provider whereas API keys or oauth flows work everywhere.