NSA Kubernetes Hardening Guidance [pdf] (opens in new tab)

> and encryption to protect confidentiality

Probably the hardest part about this. Private networks with private domains. Who runs the private CA, updates DNS records, issues certs, revokes, keeps the keys secure, embeds the keychain in every container's cert stack, and enforces validation?

That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete. How many teams are actually going to do this, versus just terminating at the load balancer and everything in the cluster running plaintext?

Who scans the vulnerability scanners? Genuine question. How does the community/ecosystem solve this problem of auditability?

I am mostly non technical person but why do we need to resort to firewalls etc. if we can employ UNIX like file permission system for network access? Wouldn't it be awesome if we can allow any installed software to contact ONLY whitelisted domains? Of course this excludes web browsers but you get the idea.

How about our mainstream OSes incorporate that kind of permission system similar to what we have in mobile OSes already have today?

> - Use network separation to control the amount of damage a compromise can cause.

> - Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality.

Are we still on this? Why isn't anyone pushing for zero trust? A concept made significantly easier to achieve thanks to container orchestration.

I do all those things in the pro version of my RBI (remote browser isolation) product, but i don't use k8s.

- Scan for vulns and misconfigs: I regularly update the underlying distro images, and use security scanning software to monitor dependencies, and regularly update them.

- Run with least privilege: I create a separate, temporary user account (no login, no shell) for each browser and service which has no elevated privileges, as well as run that browser and its service in a group and cgroup that restricts disk, bandwidth, CPU, and memory using block quotas, cgroups, tc, iptables and active monitoring and termination.

- Use network separation to isolate: RBI is basically a network isolation layer between the client (where the human interacts), and the server (where the browser actually runs.) I also don't have any privileges (service accounts, SSH keys, trusted IPs) on any of the machines and they're all single tenant and run inside GCE.

- Use firewalls to lock down connectivity + encryption: I use GCE firewall rules and iptables drop rules to block access to GCP metadata endpoints, as well as to other machines in the subnet. Also, every network request is encrypted (HTTP is https/TLS, WebSocket is wss/TLS, WebRTC is encrypted by default).

- Use strong auth to limit user access: For running the processes I use temporary users. For persistent browser sessions I use persistent users (either system native, or in a DB, always with bcrypt salted hashed passwords). For SaaS and resource control I use high entropy random API keys between each service layer. But I could improve my game for keeping secrets out of private git repos and separating code and config, ideally automatically. I could also improve my game to limit administrator access (right now I just have a single role, with God power, but I should create an admin role with power limited to a project, ideally even on a per-customer level).

- Use log autditing: I do this, but only manually, using various grepping and inspection of various logs, including last and lastb, as well as the service internal logs. This is likely something I could improve as well.

- Review all k8s settings: I don't use k8s or docker, just run services in this custom sandbox on GCE instances. I see that as both a way to limit attack service and complexity as well as minimize some overheads for maintenance and performance. In the longer term these things are worth exploring.

Thanks a lot for the TLDR. For more info on my RBI work check out https://github.com/i5ik/ViewFinder

We've actually already moved the official guidance from PSPs to OPA and that's what the primary DevSecOps reference implementation has used for about two months now.

"We" being the DoD, but our guidance is the NSA guidance. I'm not sure why it hasn't made it into the policy pdf, but the actual official IAC has been using OPA since April.

> Some useful guidance here, although worth noting that some of it is a bit dated.

Is there any digital security guidance from the feds that doesn't apply to? :)

In-tree replacement is coming in v1.22...as in, just a few weeks away. It uses admission controllers, just like OPA/Kyverno et al, hence the current guidance to use one of those.

PaaS solutions can't cover everything that PSP was covering though.

I applied for a job that wanted someone who has experience with SAML. I've actually written my own hobby IDP, and I can diagram the handshake off the top of my head. I've spent a lot of time learning how to write custom decorators to handle access restrictions. I failed my interview because they wanted me to leetcode some shit with 3d geometric volumes. I'm sorry but what does this have to do with SAML or security?

If you can't reverse a doubly linked binary prefix tree in O(1) then how can you be trusted with security?! /s :(

Consider the typical company is running servers/instances that haven't been updated or rebooted in 6 months to 3 years. Never mind the multiple year old software dependencies in their apps...

You are correct. Especially at big companies, programmers program and security is just some rules dropped on them from above.

You might be playing the long game. I think a CTO might benefit from knowing both app dev and security.

What protects the companies and freelancers who write these insecure systems from liability? Is it just a blanket “we are not liable” clause in every contract?

What do you mean prioritize leetcode? Do you feel that roles within information security require you to write "leetcode"? What even is leetcode?

The DoD maintains its own registry of hardened container images they call the Iron Bank. I guess they can't issue guidelines to the general public that you should use these, but the DoD has to use them. Which kind of sucks, because they may be hardened, but they also break all the time because the people responsible for hardening them can't possibly understand all the myriad subleties involved in building and deploying software packaged with dependencies in the same way the actual software vendors do. They make some serious rookie mistakes, like just straight copying executables out of a Fedora image into a UBI images, which works perfectly fine when a brand-new UBI release happens and it's on the same glibc as Fedora, then immediately stops working and all your containers break when Fedora updates.

there are good free/oss container scanners. check out Trivy.—no reason not to use one.

This is a great point. And containers don't even really exist in the first place, so really there should be (at least one of) a family of docs about securing the various namespaces, cgroups etc in modern Linux releases, and a doc about how to secure them in combination with each other.

You don't use this guide as a bible but take it into account and compare with other common security advice in the field. If you get similar results it most likely a good list of advice.

This isn't for regular people, this is telling third parties what they need, in order for them to try to sell something to the nsa.

How did you (/they) come up with the name Polaris?

> What yields the lowest risk - spending a ton of time hardening one cluster, or building multiple clusters to reduce the blast radius of bugs and misconfigurations?

Not sure this is a valid dichotomy.

If you are spinning up multiple clusters, you are presumably doing so in an automated fashion. If so, then the effort of hardening is very similar. It doesn't really matter where you do it.

Multiple clusters may have a smaller blast radius, but will have a larger attack surface. Things may be shared between them (accounts? network tunnels? credentials to a shared service?) in which case an intrusion in one puts everyone else at risk.

You can't skip "spending a ton of time hardening one cluster" anyways.

Having multiple clusters may help reduce the blast radius of _certain_ attacks, to some degree. However, managing multiple clusters is a lot more difficult than managing one, and you will potentially replicate bad practices, vulnerabilities to multiple places and increase maintenance burden.

If I could go back, single cluster. Any benefits you get from going multi-cluster can be achieved by configuring a single cluster correctly.

At our very large org we do both. At least two clusters per region to isolate platform changes, all hardened to the same standards using automated tooling.

with the usual caveat that it depends on your threat model, I'd say that having separate clusters is likely to provide better segregation.

Of course that reduces the benefits of Kubernetes from a cost perspective, and increases administrative overhead, so it's a trade-off.

Hmm... seems to call for a tool for a cluster-of-clusters.

If "kubernetes" is Greek for the ship's pilot, what's the name for the captain or maybe the admiral of the armada?

Both for different reasons!

what about tooling that hardens it across clusters?

In general the NSA functions more like 2 agencies, one focused on the "red" side (hacking, breaking crypto, sigint stuff) and one focused on the "blue" side (protecting US assets from being hacked, developing better/new crypto, providing guidance on security).

Both sides are good at their jobs and for what it's worth, my understanding is that the blue side really does want to keep your shit from being hacked.

They've been doing that for at least a decade, but probably quite a bit longer. Here are their hardening guidelines for RHEL 5, from 2011:

https://apps.nsa.gov/iaarchive/library/ia-guidance/security-...

They have similar guidance for Windows, web browsers, industrial control systems, etc.

For what it's worth, SELinux originated from the R&D Labs of the NSA.

> Not sure I’ve ever read the NSA providing hardening guidance on anything before.

The NSA made SELinux, SHA-1, and SHA-256.

SHA-1 was specifically a slight change to SHA-0 that was unjustified at the time but over the next 3-5 years some attacks on SHA-0 that SHA-1 was not vulnerable to surfaced.

It's fine to trust them right up until they give you a magic number.

I used them back at lockheed as early as ~2005? Although they were mostly around hardening BSD IIRC... (which became SElinux? I can't recall) and at the time, they were really "best practices" (things that you want to make sure you have done if you expect to pass any sort of audit (SOX, SAS70, etc).

Sarcastically, we would say "they already have back doors in everything, they just don't want any other Bad Actors getting in their yard"

If you take the name National Security Agency at face value, it makes sense.

It's perfectly possible, laudable even, to read things with a healthy dose of salt and still expand your understanding.

I think this document gives a good general overview, often missing in the fast-paced, crowded and noisy Kubernetes landscape.

Like many large organizations, The government has many groups, often with many of them working to some extent against each other.

AES also has been blessed by the NSA, and I bet you use that extensively too - if you want to or not?

Reasonable. But then again they did come up with selinux and I haven't seen any backdoors in that.

not sure why you are downvoted. Completely agreed.

Keeping 0-days for yourself and making security standards weaker, will just weaken your standing.

Even if this information might be useful (and without backdoors or bad advice), I just cannot trust them (so I won't click on the link).

In terms of security, I trust the hacker community much more (going by ccc or other groups advice is definitely better).

Just search for ‘* awesome list’

https://github.com/magnologan/awesome-k8s-security

(Unaffiliated with above, just popped up for k8s hardening awesome list)

> We all know it's the National Insecurity Agency[0], and that the NSA hoards & stockpiles 0day. They very rarely release tools and research papers designed to strengthen our IT infra, since they sit on so much 0day. There's no balance.

Well if the NSA does have loads of 0day then it's still better for them to give good security advice to strengthen infra, because it will limit the access adversary's have while they still have all the 0day's anyway.

i.e. they are advanced enough to not need to walk through an open door, so they might as well encourage others to close the doors because that will increase national security (while presumably not limiting their own access).

One of their missions is infrastructure security of nationally important assets. Usually this is military stuff. But think power grids, etc… NSA ironically puts out some good security stuff. Their “manageable network plan” pdf is a must read for anyone hacking to wrangle a new environment, even if it isn’t followed by the owners of said environment.

I’ve been recommending it to various localities after my security assessments for years now. https://apps.nsa.gov/iaarchive/library/ia-guidance/security-...

Have absolutely zero background knowledge here, but just to be pendantic your argument is structured as a logical fallacy [1].

While we maybe could estimate the relative sizes of the groups you mention and compare them relative to each other to guess the strategy/policy/tactics it's not clear that would be accurate; or maybe we could infer based on some heuristic or metric (like budget being a proxy for headcount), and even then it's not clear how certain that guess would be, so it's not obvious how "we all know" it's 99/1 vs 50/50, vs any other permutation.

Push come to shove would probably agree with your premise and conclusion, and really have no idea, so apologies for being nitpicky; without a background on the technical details it's likely I'm wrong.

[1] https://www.logicallyfallacious.com/logicalfallacies/Alleged...