How do you do it?
This is not a core of the business. It should be easy to use and configure so a paid service would also be good.
Consider the the following typical situation:
1. Small startup with about 10 technical people.
2. One or two are responsible for the infrastructure in the cloud like AWS, Azure, ...
3. There are also other third party services like Cloudflare, Datadog, ...
4. Everything (AWS, Datadog, ...) is managed with IaC tools like Terraform, Pulumi, (Ansible). These tools need secrets to work. The simplest way for giving the secrets to the tools is via environment variables or .env files.
5. People work on their own devices. So security cannot be absolutely guaranteed. So I guess MFA and generated temporary tokens should be used if possible. To generate temporary tokens the secrets management service has to work e.g. with AWS.
6. Optional: It would be good if Terraform can also be run in the CI/CD pipeline but only after confirmation of one of the 2 infrastructure persons.
7. Optional: It would be good if developers get credentials to setup small test environments in AWS. You can create IAM roles that only allow to create these, but you still have to manage the secrets for these.
8. Bonus: How to manage non-technical secrets, e.g. credentials for web shops to order supplies? Multiple people would have to order something. Store it all on a confluence page?
1. Application / service to service secrets should be handled either by Vault or AWS Param Store. Both are good. AWS Param Store is easier but less feature rich. Ideally, your Vault instance would have the database engine which allows for automatic rotation and password management of databases like Postgres.
2. Developers should only have access to AWS through SSO using something like Okta. Okta has MFA through it's own authenticator app (not through SMS).
3. Password sharing should be handled with something like sops or if you need a more lightweight solution - Password Pusher. Sops works really well for hiding credentials within code that needs to be deployed. A team managed 1Password instance works well too.
4. Terraform should inherit a role that gives permissions to run against infrastructure. That role should be short lived and limited to running in a CI/CD environment (preferably on your own runner if that's possible).
5. You should have a separate AWS environment for running test projects. You can do this virtually (localstack) or physically through having a separate AWS account. I prefer the later since it's the real thing and implements all the APIs. Localstack can be decent though. If I had to setup an organization now, I would have a minimum of 3 AWS accounts. One for production infrastructure, one for staging environments and one for test environments. I would also add in AWS Budget Alerts and Cost Anomaly detection to prevent huge spend on it.
Or for something more lightweight (though it sounds like you're just about getting to the size where something more instrumented is worth it), you can have a git repo with secrets encrypted with individual keys. (git-secret and pass are two of the more popular ones here)
> 8. Bonus: How to manage non-technical secrets, e.g. credentials for web shops to order supplies? Multiple people would have to order something.
Probably using a separate system. Bitwarden, 1Password, KeepassXC.
https://bitwarden.com/products/secrets-manager/
For AWS definitely use SSO with short lived credentials and proper landing zone setup.
Each user has their own vault of personal logins and related items, then various groups can have shared vaults with shared logins.
Using it with almost 100 people company wide.. working well so far!
- A combination of sops[1] and AWS Parameter Store might be a good candidate for (4)
- (5) can be solved with AWS SSO / OIDC (also: platforms that offer OIDC, usually support that feature for free, and a variety of them can be configured via Terraform)
- For (6), there are solutions like Atlantis[2] for Terraform, but I am not familiar with alternatives for Pulumi and Ansible. Might be a good idea to spend some time and build a pipeline with whichever CI/CD platform you are using (bonus: GitHub Actions supports OIDC auth with AWS as an IAM role[3], no secrets nor credentials required!)
- For (7), you can create a separate AWS account and manage it with AWS Organizations (or not)
- For (8), 1pass[4] for any credentials that requires username/password combo (non-tech/breakglass account). Groups can be created to limit certain credentials to subsets of users
[1] https://github.com/mozilla/sops
[2] https://www.runatlantis.io/docs/repo-level-atlantis-yaml.htm...
[3] https://docs.github.com/en/actions/deployment/security-harde...
- About 5, https://github.com/99designs/aws-vault is super handy to keep AWS credentials encrypted.
- Use full-disk encryption as a team policy.
- About 7, it is likely that this is solved by having preview environments so that developers don't need to manage infrastructure manually.
- About 8, 1password shared vault works nicely for us.
- Where possible enable SSO, even for AWS or other cloud services (https://blog.leapp.cloud/how-to-saml-federate-your-aws-accou...). Here an example on how to do it with GSuite and aws
- For all your employees I can advice you Leapp as open-source project (https://github.com/Noovolari/leapp). It solve mayor of the problem listed here:
. Responsable for the AWS infrastructure get the IAM Security standard on track (short-lived credentials, MFA, IAM Users secured, rotation of credentials and access to the console with a set of secured credentials, and generation of Azure ceredentials)
. temporary tokens are mandatory, so let a project like leapp locally manage IAC credentials for you.
- Is not too strictly to manage pipelines only after confirmation? You can create a specific role for managing a specific policy to deploy your pipelines. Here an article on how to start organizing your AWS account as a startup (https://blog.leapp.cloud/aws-multi-account-strategy-explaine...)
- "People work on their own devices.": that's why Leapp has been created, check it out.
- Optional: "It would be good if developers get credentials to setup small test environments in AWS." how anout creating a sandbox account for them? - "Bonus: How to manage non-technical secrets" even here, Leapp is a desktop app and is vastly used also for accessing to EC2 instances even for not technical people
I hope this guide can help you!
At my current gig we're using doppler[^1] (no affiliation) for application secrets. We're using doppler with their kubernetes operator which supports auto-rotation on secrets. Secrets are set as "env variables". So far, doppler has not suffer big outages or we did not notice, because the operator will keep working even if their API is down - of course you won't get updates. Access control could be more _fine grained_ and they added secret auto-rotation option recently[^2]. We don't use that yet.
I've been a happy 1Password user (no affiliation) and we use it company wide to share user secrets. 1Password support CI/CD integration IIRC, so in theory should cover most use cases.
If you can pay for AWS Vault, the terraform integration comes out of the box. However if you're a small team running vault might be a daunting task and you're inserting another SPoF.
There are many open source application secrets tools that you could check out though. In the past I had great experience with credstash[^3]. Credstash is a really simple and secure open source solution that is based on AWS KMS, IAM and DynamoDB. Costs pennies to run for medium size deployments. Once you setup and document the way to use it, it's really easy. The downside is that as a tool is pretty _raw_ you have to build things like "secret generators", etc. But combined with a slack bot can be a really powerful, secure, cost-effective self-hosted solution that doesn't require maintenance (as opposed to solutions like Vault).
If you have specific questions about any of the above tools feel free to drop an email. I'll happy to answer questions.
[^1]: https://www.doppler.com/
[^2]: Auto-rotation is complicated because you need to integrate the auto-rotation with external tools yourself most of the times. There are Hashi-Vault modules for SQL DBs but not for Mailgun or CloudFlare for example.
LastPass, 1Password, Vault and Bitwarden, all these are cloud-based password manager that offers secure storage and sharing of login credentials. they also includes features like MFA and the ability to generate temporary tokens.
As for how to do it, I recommend evaluating each of these tools based on your specific needs and choosing the one that best fits your requirements. In general, you should look for a tool that offers strong security features, easy integration with your existing tools, and a user-friendly interface.
I moved off of their service because I could no longer trust it.
All secrets are in a file on there. I make it a file of exports I source before launching the app. Backup the file in 1pw.
Secrets get deployed along with the app to a server also secured with ssh and a firewall/VPC.
Maybe doesn't scale to an org chart, but a scrappy startup of 3-5 with some contractors and it works well. This method allows you to literally have a single VPS prod host and a single VPS deploy box. That's $10 and super simple and completely cloud-agnostic.
1Password may be easier to setup with easy CLI/UI access for secrets as well as in your CI pipeline. It's paid but I love the ergonomics of the system. It's what I do for all my personal projects.
Gitlab CI/CD secrets is what we use at my startup. The most annoying thing with Gitlab secrets is that you need to give maintainer access to use secrets in local scripting, but we've been able to work around it for now
For non-technical secrets, most organizations have a password manager (1password has been the go to at my previous two orgs)
1) Make the perfect pitch for your startup: https://www.ratherlabs.com/post/how-to-make-the-perfect-pitc...
2) Network Effects: https://www.ratherlabs.com/post/network-effects-how-can-it-i...
Devs, once assumed a role, download the secret env files from secretsmanager.
You might see where I'm going with this, you have two systems to store secrets in, one for general use and one for programmatic use. It's not ideal but pragmatic and do consider if your in a security sensitive domain, then you would want to customise this process e.g. by only having one storage system and shorter access etc.
Someone of the two infra people would have the ability to assume a role/get a set of expiring "deploy" keys that they would paste into CI To start a deployment (still in ci just doing stuff that CI normally doesn't have permission to do)
