Security Is Not An Encryption Problem

Not in the cloud, anyway

In this post I cover:

• That I really do like encryption, so you don’t get too mad about the rest

• That encryption at rest and in transit requirements don’t really make sense in cloud environments

• A deep dive into the various ways of doing encryption in cloud environments and how they’re all silly

• Encourage the security community to please do something more productive with all of our time

Encryption

Encryption is a wonderful thing. It feels miraculous; the only reason we're able to have any secret and trusted communications over the internet is that we can math in particular ways.

We're also, as a community, used to pushing encryption everywhere we possibly can. This is almost always for good reason. Having TLS and HTTPS everywhere is such an accomplishment, and it still befuddles me that it took so long and there was such odd pushback on it. We've come so far with encryption at rest that you can be pretty confident that the loss of your device doesn't mean data loss. End-to-end encrypted messaging is so well built that our government friends are wailing about various things to try and fearmonger us into breaking it. We've come a long way.

We've been drinking the "Encryption is Good" Kool-Aid for so long, though, that it's now starting to become a useless control we demand without knowing why. This is especially prevalent in cloud environments, which this post focuses on.

Encryption at Rest Isn’t Useful in the Cloud

If you ever work at a B2B SaaS company, one of the things prospects want to hear most is that all of your communications are encrypted at transit (you're communicating over HTTPS or TLS so that no one besides you and your intended recipient can read or modify the message) and at rest (you are encrypting data with AES-256 when storing to disk, so that no one besides you, the data owner and key holder, can access the data).

Encryption at rest is pretty trivial. Here's what that entails in AWS for S3:

Besides the humor of this being a literal checkbox, it’s a silly control in a cloud-native world. Encryption at rest prevents someone with access to the physical drive or file (but not the decryption key) from being able to access the data. This is great for say, laptops, but think about it in cloud land. All AWS storage encryption is decrypted seamlessly when you request the encrypted data. It doesn't stop me, someone with an IAM policy giving me access to an S3 bucket, from reading the data. It just stops people from getting value from stealing the drive from the data center. For this control to have value, Tom Cruise needs to be in my threat model:

I'm going to be honest with you, if someone breaks into AWS us-east-1, evades security, figures out which drive is mapped to my S3 bucket, finds it in the depths of the datacenter and runs off with it, I'm… kind of fine with that? You do, in fact, have to hand it to them. That sounds like a fun incident report to write and disclose, if a bit of a weird RCA. That's very much a Mossad thing to do, so I'm just happy they decided not to use the uranium phone.

NOTE TO ENTERPRISE PROSPECTS AND CLIENTS: I care very much about your data security, which is why I only ever use highly secure data centers with military-grade physical security, and subscribe to several threat intelligence and flight data feeds tracking the exact whereabouts of Tom Cruise and George Clooney at all times.

Sure, you can argue that encryption at rest provides defense in depth; some segments of AWS could be compromised, and having encryption at rest turned on could help. If you're mature enough for that to matter, and aware enough about AWS's internal architecture to effectively threat model when it would be valuable, great! You're probably just Amazon?

This isn't billed as a control for highly mature organizations, though. It's billed as a foundational control by CIS, compliance frameworks, cloud configuration scanning tools, etc, above any sort of IAM policy least privileging, network isolation, or patching programs. That's insane.

This is primarily a checkbox; it's pretty easy to implement and require in your infrastructure-as-code linters and AWS Config or whatever other cloud security posture management service you probably overpay for. AWS has made this dumb control very easy. Bless them. You may as well check the box!

The ease of implementing a useless control doesn't excuse the degree of attention it gets in the community, though. I spend a depressingly large number of my calls with prospects saying that, of course, we encrypt everything at rest. I know many others do too. It'd be a boon just to get that time back.

Customer-Managed Keys

There's a sub-set of encryption at rest that some enterprise prospects will care about called customer-managed encryption keys (CMEK). In this context, the customer is you, the AWS customer, not your enterprise prospects (confusing, especially for reasons we’ll get to later).

For encryption at rest in AWS, it's just a checkbox; you press a "please encrypt this for me" button, and they do, and Tom Cruise cannot hurt you, no matter how many sysadmins he charms his way past with his dazzling smile. If you're sane and use AWS-owned keys for this, AWS does everything for you. You press the check box, the key is stored somewhere in AWS KMS behind the scenes from you, requests for encrypted data (from IAM entities with the right policies) will have the data seamlessly decrypted on reads and seamlessly encrypted on writes. This is automatic and, more importantly, entirely free.

You also have the option to use customer-managed keys. These are keys you define, generate, and manage in KMS. You can configure all of AWS's encryption at rest tooling to use this managed key.

AWS KMS is backed by a bunch of hardware security models (HSMs), which, for our purposes, are magical black boxes that do cryptographic operations and key management without it being possible to extract the actual keys to run off with. AWS uses the same backing HSMs to power both customer and AWS-managed keys¹. The underlying architecture and cryptography are the same for both AWS-managed keys and customer-managed keys, and you get the same result of your data being encrypted at rest by keys stored in HSMs that no one, not your employees, not AWS employees, can directly access.

So, what's the point of using customer-managed keys? Your enterprise prospects probably want to know either that they have their own special key or that it is rotated with every rising of the blood moon over their headquarters rather than annually. Some things are different when using them:

• You get to see the key in the KMS console, bask in how pretty its name is, and press the rotation button if you want to.

• Note: Rotating a key in KMS isn’t real rotation.² KMS will keep every version of the key for all time, and use a past version of the key to decrypt objects encrypted with that past version. It just doesn’t let you encrypt new things with old keys.

• The key is only used for your account. This is pointless because, again, the keys can not leave the HSM, and you're trusting the KMS application logic to apply the cryptography operations correctly for either the AWS-owned or customer-managed keys.

• It's not free. You get pay-per-use fees on every operation. This can add up.

• They count against your KMS API quotas, which you can hit pretty quickly.

• You can accidentally remove policy access to the intended datastore and take out your application. New failure modes are exciting.

This is basically the cloud version of an NFT? Something you pay money for an entry in a datastore (the KMS key list) that provides no value, but you hope someone else (your enterprise prospect/client) will pay more for later on. AWS themselves can’t come up with a real reason you would need to rotate their keys; from their blog announcing fancy new rotation features:

Although we advise that key rotation for KMS keys is generally not necessary to improve the security of your keys, you must consider that guidance in the context of your own unique circumstances. You might be required by internal auditors, external compliance assessors, or even your own customers to provide evidence of regular rotation of all keys.

The only unique, individual circumstance AWS can come up with for key rotation is your customers made you do it. Chimps!

More pragmatically, a KMS key behaves less like an encryption key and more like a second resource in AWS that you need access to in order to read the data. It’s an IAM policy object to provide access control, not an encryption key. You never really interact with the encrypted data; I'm not sure how you even would. If you can access the data object (and the key) via IAM policy, AWS will seamlessly decrypt it for you and return the decrypted version. If you don't, it will return an HTTP 403, not the encrypted content. This can provide some minor defense in depth for your IAM policies, but no one (compliance standards, government regulations, enterprise prospects) is reasoning about it this way; they’re acting like it’s really encryption.

Customer Customer-Managed Encryption Keys

There is also the option for a customer-managed encryption key that is managed separately by your enterprise customer, in their own KMS instance in their own AWS account. You will spend the first several minutes on a call with your enterprise prospect figuring out which type of customer-managed key setup they actually want.³

In the prospect-managed key situation, they grant you IAM access to use a key in their account, and your S3 or RDS or whatever will use it as seamlessly as if it was your own key in KMS. This has the following effects:

• Your customer gets to see the pretty key name in their AWS console and press the rotation button whenever they want to.

• They pay the per-use fees instead of you.

• They get to accidentally take out access to the key or overload the KMS API, knocking out the application you’re running for them. This is nice in that it’s their fault, but it’s bad in that they will probably still yell at you about it.

• They can intentionally take out your access to the key and remove your access to the data it encrypts.

That last point actually does something! In a technical sense, this is cool! The crypto nerd in me loves it. In a value-provided sense, if you do this, assuming your SaaS provider (or AWS) stores your data in the right places that are only encrypted by your controlled keys, you have ownership of your data. You can turn off their access if your relationship is souring or if they are being hacked.

There are some cons with this, though:

• That assuming your SaaS provider stores your data in the right places bit is doing a lot of heavy lifting. You still have to trust your SaaS provider isn't just mirroring your data to an S3 bucket with a normal key or something, let alone unintentional fuckups like logging all your data in verbose debug messages.

• You also have to trust AWS not to do any of those fuckups.

• If someone hacks your SaaS provider, they've probably exfiltrated all of the juicy bits of your data before you find out about it and can cut off access. If someone hacks AWS, they’ve probably stolen all your data from all of your own systems and any of your other SaaS providers, or from the NSA’s copy of it on GovCloud.

Still, this actually does something. There are a lot of assumptions that have to be true in order for it to do so, and they probably aren’t true in most cases, but it’s still real. That said, of the companies that have asked me about this feature, none of them have cared to confirm that my IAM policies aren’t horrifying. So this is still a chimp-like fixation on encryption at rest above far more important things, rather than being highly mature security orgs mitigating the last bits of their threat model.

External Key Stores

If your enterprise prospect is silly but very sophisticated, rather than just wanting you to use the AWS customer-managed key feature, they may wish to supply their own key management service to back KMS for their encryption keys. This is a relatively new feature; AWS released it at the end of 2022.

The KMS External Key Store allows you to provide a service that you hit external to AWS for KMS to use to do the cryptographic operations. Instead of KMS proxying operations to a backing HSM that AWS controls, it proxies operations to the external key service that someone else controls. You can use keys managed in this way as if they were normal customer-managed keys within AWS, like to back encryption of your S3 buckets and databases. In this context, your enterprise customer controls the external key service and the keys hosted there. This has the following implications, in addition to the customer customer-managed keys above:

• You now have longer latency on key operations.

• You have to manage a new external key service. This is either a different vendor that you, for some reason, trust to hold your keys more than AWS, or an additional system you have to manage. You now rely on both it and your SaaS provider to maintain uptime to use the service. New failure modes are exciting.

This is really cool. From a crypto-nerd perspective. I mean, I cannot fathom a world where you trust AWS to host and process all your data, but not trust KMS to store the encryption key. So I have no idea why I’d ever spend budget implementing it. But still, cool!

Encryption In Transit is Useless in the Cloud

Remember how I said HTTPS is wonderful everywhere? That's true in all public networks, and in traditional networks, where compromise can allow the viewing of traffic traveling between other machines. Gotta protect that data. It's also sometimes a checkbox in AWS:

Encryption in transit protects you from two things:

• A compromised machine on your network being able to view or modify traffic to unrelated machines

• A compromise of the network itself providing access to traffic between machines

On the first piece, this is not possible in AWS (and other cloud providers) without much prior effort. The world of popping one machine and getting visibility into the whole network is gone; this is no longer part of the threat model. Instead, you use the SuperAdmin IAM Role assigned to the EC2 instance to take over everything with API calls. Different!

On the second point, this doesn't make sense in a cloud environment. Sure, you can argue that it provides in-depth defense in case the internal network of your cloud provider's data center is compromised.⁴ However, that's highly correlated with everything else being compromised. I'm not sure of the scenario where just network compromise occurs without that threat actor being up in your instances anyway. Tom Cruise is in the data center; he probably has access to more than one switch.

Conclusion

With both types of encryption in the cloud, it's not that we shouldn't turn it on. it's that we spend far too much thought and effort on it, to the detriment of far more critical things. We need to de-emphasize how important encryption in the cloud is in our collective mindshare. We treat it as a top-10 problem; I don’t think it’s even top-200.

If an AWS account has a bunch of IAM users with admin permissions or iam:*, and with a bunch of data stores without encryption turned on, your auditors, customers, and cloud management platforms will be up in arms about encryption. I get asked about encryption in transit and at rest all the time by clients. I'm very rarely asked for details about my IAM policies. One has the potential to ruin your week! It's not encryption.

I'd Love To Hear From You

Do you agree? Disagree? Intensely? Do you have any other feedback for me? Please leave a comment below; I'd love to hear it!

1. This obviously glosses over encryption key hierarchy in AWS. In reality, just the root (or wrapping) key is in KMS, and it is used to encrypt and decrypt various data keys that AWS S3 creates on the fly (see https://docs.aws.amazon.com/amazon-s3-encryption-client/latest/developerguide/concepts.html). This implies that no matter what method of key storage you have, you’re still trusting AWS to appropriately delete the decrypted data keys after using them. This is another reason that customer-managed keys are silly.

2. You want it not to be real key rotation, because real key rotation is horrendously painful in most contexts. You technically can set KMS not to store past versions of keys, but you’ll probably regret that.

3. Horrifyingly, they sometimes want to email you a specific key they generated they want you to upload into KMS a customer-managed key that you control. I have no idea why this is a thing they want, but it’s come up repeatedly. Maybe they think the key will remain loyal to them if I try to misuse it?

4. You can and should argue mTLS is an excellent service isolation feature. I agree! I really love mTLS. I love solving security problems with certificate trust hierarchies, and I love that service meshes are making that easy now. The value here is in that service isolation, not in the communication channel security, so I feel that’s out of scope here.