The entire point of encryption at rest (on the cloud) is that when any of the following happen you have nothing to worry about.
1. A machine/disk is rendered inoperable and can't be wiped.
2. The data stream coming off of a disk cluster is tapped.
3. An employee steals a disk or is lost.
4. An actor violates their account segmentation and can read raw data from segments of YOUR sectors of the shared disk.
5. SSD firmware goes bad/gets hacked and starts returning incorrect sectors of disk.
6. Memory pointers go bad and return sectors from the wrong area of the disk.
It's incredibly naïve to not use encryption at rest on AWS with how incredibly easy and problem free it is to deploy.
As I mention elsewhere in the thread, I use encryption, but I don't consider it to be of high value compared to other security mitigations one can spend time on.
As mentioned in the blog post, setting up encryption isn't always easy and problem free though.
Regarding 1, disks are physically destroyed if they are inoperable. Thanks for sharing the list of potential problems, it's always interesting to see how other people think and what they worry about!
'Supposed to be' is missing from this statement. We have seen many vendors over the years say 'we destroy disks' only to have a disk show up in the wild.
Where is it you draw the line?
Do you bother with Spectre mitigations since Amazon policy is to deny service to those who would attack you?
Do you bother requiring authentication on your database since policy is to use a closed VPC?
Hell, we haven't seen any man in the middle attacks lately, let's just drop SSL because you know your customers are wired and trust the endpoint networks.
SSL is the best example subject to the same common failures (including expiration, key loss, downtime to initially deploy) as you describe disk encryption migrations.
Encrypting disks is mandatory and there is zero justification otherwise.
It shocks me that this isn't on-by-default in AWS like it is on GCP.
By now that's moot so I'd assume they'll set it by default for new accounts at some point but there's a regional option which you can turn on by default:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncry...
If you use tools like Security Hub with the Amazon Foundational Security Best Practices suite of controls enabled there's a specific check for that setting:
https://docs.aws.amazon.com/securityhub/latest/userguide/sec...
I.e. how do you even know it's working and your data is encrypted? Just because API says so, doesn't mean it's so. There is no way for you to verify. Is there? Unless you encrypt on your end, you can't be sure.
I know people who work there. I've read the infrastructure designs. I know Amazon audits themselves. I know AWS has third party audits. I know AWS has government and enterprise users who require their partners be legally liable for improper implementation. I do not subscribe to conspiracy.
Also, sometimes there are mistakes with decommissioning old drives, and you wouldn’t want your data discovered in a landfill somewhere.
When I dug deeper, they had wiped the SMART data and the partition table, but it was absolutely full of readable data. I found clear text server logs indicating that this drive was in a backblaze center for several thousand hours.
I assume aws is similar
Adding full disk encryption takes time from other projects and makes the system more complex. That equation needs to pay out. In all likelihood, the reason your data is going to get stolen is a privilege escalation in your app code or a bad actor on your team. Rogue AWS employee swiping your particular hard drive in us-east-1 is way down the list. Full disk encryption does nothing for the first two vectors.
I think compliance programs are oriented around pushing companies into complex/expensive system designs thinking that is a proxy for a secure system.
Usually you inherit an infrastructure, and it's usually not set up in this way (in my experience) and then there is a lot of work to re-encrypt the data in order to use KMS rather than the default key.
> it is typically standardized in an org
I have still not found any SCP I can set that prevent the use of the default key and enforces KMS. If you have one, I'd be happy to take it! If you mean "standardized" as in written on a paper, I'll rely on wishful thinking because people make mistakes or just don't know about it even if it's a standard.
Compliance has good sides too though. For example, they force you to think about areas your intuition might otherwise not have gone, so I don't dismiss them but sometimes it makes you spend time on less than optimal things in order to stay compliant.
This sounds like it's coming from someone who's forgetting that all of our data in "the cloud" resides in physical buildings throughout the world, which are all high value targets for physical attacks.
I don't think it's an antiquated idea to protect against physical data center attacks. It's best practice.
Could a data center break-in happen? Sure. Is it likely? No. When used in a data center, FDE is mostly useful when media is being transported or disposed, as an extra layer of protection.
I’m not saying it shouldn’t be used, but when comparing using FDE in other situations vs. a data center, in a data center the physical risk is far lower.
If they do, I trust them to be good stewards of their keys. If not, then why not? I don't know about the enterprise part of the market, but I know consumer SSDs come with encryption permanently enabled in the firmware.
Where is the encryption key? It seems that it must be in the firmware itself. Presumably it would be possible to find this with enough effort.
The people with physical access in AWS datacenters are not the same people who have access to the encryption keys.
In fact, it's likely much more complex than that. The people with software access to the machines very likely don't have access to the key storing system (and definitely not the hardware).
This means that the number of people it's possible to bribe to get access to your data is much smaller than if you can just bribe a DC employee to either smuggle the disk out, or make a copy onto a thumb drive.
I'm not saying the number of people who could "break glass" and read anyone's data is zero. But it's at least an order of magnitude fewer than the hoards of people employed to swap hard broken drives all day every day.
And the people with this "root" access will likely be very well paid, reducing (but not eliminating) risk of bribes.
Public Cloud cloud have their shit together, but they also deal with millions of hard drives. I could definitely see a story coming out where someone finds a hard drive, some sensitive stuff on it, and just nobody has any idea how it got out. Stranger things than that happen every day.
If it's encrypted then that's another layer of swiss cheese.
But also.
> The author is not talking about getting someone with physical access.
Right. The author missed this as one of the major attack vectors that this aims to protect against. I don't think leaving out a real justification for it when saying it's snake oil speaks in favour of the author's point.
Most well paid engineers at AWS won't have access either. Presumably some minimal set would, but they would likely be pretty senior, and well paid.
But ideally you'd want one set of people with access to the keys, another with access to the data, and a third with physical access, and no overlap. That way you need three people to conspire.
But that's going to be very hard to achieve in practice. But it's not all or nothing. The closer you get to this goal the harder it'll be for someone to not just do it, but do it without triggering a tripwire from the security folks, or at least persist a log entry that if found would get them thrown in prison.
Public cloud companies are not like a small startup where everyone has root.
(sounds like twitter kinda was, according to recent reports)
[1] The trust moves to the CPU vendors instead of the cloud vendors, but if you don't trust CPU vendors then you're going to have a hard time doing anything with computers in the modern world.
(Note confidential containers builds on top of confidential hardware VMs)
I think the real issue is higher-ups think that if data is encrypted that somehow mitigates meaningful exfiltration, but just look at data breaches like Capital One and you'll see that's not the case.
The whole stealing the (correct) hard drive or concerning yourself with a host-level attack should be at the very bottom of the list as far as your threat model goes -- these are not really typical of the surface area of most deployments and you're better off focusing on the surface area such as application security, transport security, having your applications perform the encryption than relying on AWS controls, etc.
The author is saying the probability that it improves your security is so low that any time it takes away from other security related work is actually making your security worse. That’s a different threat model, and it’s probably the better way to approach security, and the author is probably correct - if he actually uses that saved time to improve security in other ways.
I have a question though, what are the laws that require encryption? I can think of HIPPA, SOC2 maybe? What about GDPR?
For instance, it greatly expands the scope of ‘I lost your data/I exposed your data’ notifications in places like California, and for those under many of those other rulesets.
Someone getting access to a repo of encrypted drive images, or someone losing an encrypted drive, doesn’t count. And for reasonableish reasons.
It’s a basic risk mitigation/blast radius limiting move.
For physical/on-prem especially, since old disks tend to ‘wander’ after retirement, and it’s a great idea to always have had them full disk encrypted to reduce the odds someone sensitive gets exposed years down the line.
Beyond the compliance requirements (e.g. CIS), it’s wrong to assume that disks are always perfectly disposed of – simply never having to worry about featuring in one of those “look what I found on eBay!” posts and notifying your customers is worth the basically non-existent cost of basic encryption. It’s something you turn on and basically never think about again - the performance hit disappeared around 2010 and it’s easy to enable globally.
Beyond that, however, there are two key benefits - the author even almost discovers one of them but didn’t think about it enough: encryption means deletion is probably fast - delete the key and you don’t need to monkey around wiping disks and snapshots.
The other, bigger, one is that it can protect against mistakes or compromises. If you share a KMS-encrypted resource accidentally, an attacker won’t have access to the data. An attacker can’t access a resource with a custom KMS policy unless they compromise the specific role which has access or do more invasive things which will trip alerts. Again, not perfect but it protects against common mistakes like the Capitol One firewall breach, which is why all of those standards started recommending it.
That was not my intention and it's unfortunate that's how it sounds. Just like most people don't need armed guards, most people don't need bank level security for their disks either. Encrypting disk encryption is usually not the most effective thing you can do to improve your security, there are usually other actions with better payoff.
AWS zeroes the disks before they're reused, so from your point of view it's instant. As mentioned in the post, deleting a KMS key takes at least 7 days (and up to 30 days, depending on your configuration.) Regarding other point, that's also mentioned in the post as "if your IAM access configuration is bad your KMS access configuration might save you," which is referring to what you mention.
> It’s something you turn on and basically never think about again - the performance hit disappeared around 2010 and it’s easy to enable globally.
This is true if you, or the one you inherit the infrastructure from, configured it correctly from the start. But re-encrypting disks, databases, S3-buckets etc is time consuming and might require downtime. So it's not always easy or cost free (in terms of labor.) I'm not sure what you're referring to with "easy to enable globally", enable what?
Thanks for sharing your perspective!
How many ways are there to do it wrong than to do it right?
This is partially true with respect to EBS (i.e. virtual disks). When you boot up an EC2 instance, the a plaintext data encryption key is loaded into hypervisor memory. As long as the EC2 instance is still running, you can add an additional EBS volume to the instance and then copy data from the disk that has the orphaned key. Even if the underlying KMS key was deleted.
The rest of this stuff feels like FUD me. Yes, IAM has identity-based policies and KMS has resource-based policies. And yes, default service KMS keys are unique per account (why would you expect otherwise?).
Implementing an encryption program takes forethought. This is true if you're hosting your own data, too. I really don't miss manually rotating drives, fighting with LUKS, and then putting drives in a fireproof safe (which was never locked anyway, so I'm not sure if it would've actually protected anything in an actual fire).
I expect it to be unique per account, but I would be happy if it was possible to share it with other accounts so one could make cross account backups (it's good practice to have a separate AWS account for backups.) Currently this requires a KMS key, which means data encrypted with the default key must be re-encrypted and that takes a lot of time and effort.
https://en.wikipedia.org/wiki/Swiss_cheese_model
AKA defense-in-depth
Relying on one control is a recipe for failure, which is why security measures work best when layered.
You don’t trust in one control, you trust that you stack enough controls that one of them works.
Just because something isn't 100% perfect in every scenario doesn't mean it shouldn't be done at all.
But I agree with your point, if you're really worried about data to the point where you don't trust AWS with encryption keys, you should self-manage your keys and manually encrypt/decrypt data without AWS KMS.
As for some of these statements (i.e. loss of key equals loss of data, re-encryption with a new key is expensive to go through an entire disk, etc.), that is exactly the point of encryption and holds true beyond the cloud. It is not meant to be trivial; it is a layer of security that trades convenience and often some level of performance.
That's pretty much the case for any type of encryption.
This blog post is like someone complaining "parameterized queries are so much more of a pain than just using string interpolation to build my queries. I'd never allow SQL injection in my app." Hubris of an inexperienced engineer imho.
This seems unnecessarily hostile. I've worked with cloud infrastructure and security for more than 10 years. If you have experience of unencrypted disks being stolen from AWS I'm very interested to hear about it.
Note, though, that I don't claim one shouldn't encrypt disks, but I consider it to be very low on the list of priorities when it comes to lowering risk. There is almost always risk-lowering actions with better cost:benefit ratio than encrypting disks in the cloud, since that risk is already so low.
If FDE is easy to do, it’s usually worth it to reduce the risk to zero.
Is this really the only vector for data leakage?
> You used the AWS default key to encrypt a database?
Don’t do that? Create a separate account to hold keys. Lock it down like you would your domain DNS or anything else with security implications that will impact your entire company.
Share granular read permissions across accounts as needed to encrypt / decrypt.
Agreed it’s time consuming to initially setup, but it’s a solved problem and can be implemented with your IaC flavor of choice (CloudFormation directly, CDK, Terraform, …).
What you describe is essentially what I currently do. But I've inherited an infrastructure that was not setup that way, and re-encrypting things has been very time consuming.
The company I'm at now use multiple AWS accounts where teams have their own accounts, and it's common for people to forget to use KMS when creating databases or similar. I might have just failed in my search but I couldn't find any way to block default keys via SCPs. If you have any suggestions for that I'm happy to take them!
