eBPF will help solve service mesh by getting rid of sidecars (opens in new tab)

> Honestly after I learned that the majority of Kubernetes nodes just proxy traffic between each other using iptables and that a load balancer can't tell the nodes apart (ones where your app lives vs ones that will proxy connection to your app) I got really worried about any kind of persistent connection in k8s land.

There can be a difference, if your LoadBalancer-type service integration is well implemented. The externalTrafficPolicy knob determines whether all nodes should attract traffic from outside or only nodes that contain pods backing this service. For example, metallb (which attracts traffic by /32 BGP announcements to given external peers) will do this correctly.

Within the cluster itself, only nodes which have pods backing a given service will be part of the iptables/ipvs/... Pod->Service->Pod mesh, so you won't end up with scenic routes anyway. Same for Pod->Pod networking, as these addresses are already clustered by host node.

That's if you're using a NodePort service, which the documentation explains is for niche use cases such as if you don't have a compatible dedicated load balancer. In most professional setups you do have such a load balancer and can use other types of routing that avoid this.

https://kubernetes.io/docs/concepts/services-networking/serv...

Whether load balancer can or can-not tell the nodes apart depends on load balancer and method you use to expose your service to it, as well as what kind of networking setup you use (i.e. is pod networking sensibly exposed to load balancer or ... weirdly)

Each "Service" object provides (by default, can be disabled) load-balanced IP address that by default uses kube-proxy as you described, a DNS A record pointing to said address, DNS SRV records pointing to actual direct connections (whether NodePorts or PodIP/port combinations) plus API access to get the same data out.

There are even replacement kube-proxy implementations that route everything through F5 load balancer boxes, but they are less known.

This is a concern only if you have ungraceful node termination Ie you suddenly yoink the node. In most cases when you terminate the node, k8s will (attempt to) cordon and drain the nodes, letting the pods gracefully terminate the connections before getting evicted.

If you didn’t have k8s and just used an autoscaling group of VMs you would have the same issue…

The complexity is an issue (but sidecars are plenty complex too), but the security not so much. BPF C is incredibly limiting (you can't even have loops if the verifier can't prove to its satisfaction that the loop has a low static bound). It's nothing at all like writing kernel C.

I don't think client libraries are the answer. If you only have one technology stack (say, Java and Spring) and only use one application-layer network protocol (say, HTTP), then maybe it's fine.

But once you have more than one language or framework, you need to write more and more of these libraries. And what happens if it's not just HTTP? What if you need to speak Redis, MySQL, or some random binary protocol? Do you write client libraries for those too? Maybe a company like Google has the scale to do this, but most orgs do not. But even then, what if you have to run some vendor-supplied code that you don't even have source for?

I agree with you that shoving more of this into the kernel isn't desirable, but libraries aren't great. Been there, done that, don't feel like doing it again. I'd rather stick with sidecars.

If you are in a position where you can do that then great. Most folks out there are in a position where they need to run arbitrary applications delivered by vendors without an ability to modify them.

The second aspect is that this can get extremely expensive if your applications are written in a wide number of language frameworks. That's obviously different at Google where the number of languages can be restricted and standardized.

But even then, you could also link a TCP library into your app. Why don't you?

I'm not necessarily advocating for the approach described in the article but it wouldn't worry me from a security perspective. The security model of eBPF is pretty impressive. The security issues arising from engineers struggling to keep the entire model in their head would concern me though.

The industry is moving away from the client library approach. This is possible in a place like Google where they force folks to write software in one of four languages (C++, Java, Go, Python) but doesn't scale to a broader ecosystem.

In a world without (D)COM, I find it's much, much harder to make common base libraries and force people to use them, especially if you can't also force limit the set of toolchains used in the environment.

In addition to whether or not all of your various dev teams preferred languages have a supported client SDK, you also have the build vs. buy issue if you're plugging COTS applications into your service mesh, there is no way to force a third party vendor to reengineer their application specifically for you.

This probably dictates a lot of Google's famous "not invented here" behavior, but most organizations can't afford to just write their entire toolchain from scratch and need to use applications developed by third parties.

It is the technically better solution IMO/IME, too.

But that doesn't work when you're trying to sell enterprises the idea of 'just move your workloads to Kubernetes!'. :)

> a good RPC library

I like that approach. If you use client libraries, new RPC mechanisms are "free" to implement (until you need to troubleshoot upgrades). It's also an argument against statically linking.

For instance, if running services on the same machine, io-uring can probably be used? (I'm a noob at this). eBPF for packet switching/forwarding between different hosts, etc.

It does feel a bit like we're trying to monkey patch compiled code but the benefits are pretty clear.

That’s great if you write all your software. If you want to use someone else’s thing then you have to wrap it in that magic everywhere client.

What if a client library does not yet exist for your language?

The big secret is that sidecars can only help so much. If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars). mTLS is similar; it's great to encrypt your internal traffic on the wire, but that needs to get propagated up to the application to make internal authorization decisions. (I suppose in some sense I like to make sure that "kubectl port-forward" doesn't have magical enhanced privileges, which it does if your app is oblivious to the mTLS going on in the background. You could disable that specifically in your k8s setup, but generally security through remembering to disable default features seems like a losing battle to me. Easier to have the app say "yeah you need a key". Just make sure you build the feature to let oncall get a key, or they will be very sad.)

For that reason, I really do think that this is a temporary hack while client libraries are brought up to speed in popular languages. It is really easy to sell stuff with "just add another component to your house of cards to get feature X", but eventually it's all too much and you'll have to just edit your code.

I personally don't use service meshes. I have played with Istio but the code is legitimately awful, so the anecdotes of "I've never seen it work" make perfect sense to me. I have, in fact, never seen it work. (Read the xDS spec, then read Istio's implementation. Errors? Just throw them away! That's the core goal of the project, it seems. I wrote my own xDS implementation that ... handles errors and NACKs correctly. Wow, such an engineering marvel and so difficult...)

I do stick Envoy in front of things when it seems appropriate. For example, I'll put Envoy in front of a split frontend/backend application to provide one endpoint that serves both the frontend or backend. That way production is identical to your local development environment, avoiding surprises at the worst possible time. I also put it in front of applications that I don't feel like editing and rebuilding to get metrics and traces.

The one feature that I've been missing from service meshes, Kubernetes networking plugins, etc. is the ability to make all traffic leave the cluster through a single set of services, who can see the cleartext of TLS transactions. (I looked at Istio specifically, because it does have EgressGateways, but it's implemented at the TCP level and not the HTTP level. So you don't see outgoing URLs, just outgoing IP addresses. And if someone is exfiltrating data, you can't log that.) My biggest concern with running things in production is not so much internal security, though that is a big concern, but rather "is my cluster abusing someone else". That's the sort of thing that gets your cloud account shut down without appeal, and I feel like I don't have good tooling to stop that right now.

I'm sure someone will write leftPad in eBPF any day now.

We're a global platform that runs an intra-fleet WireGuard mesh, so we have authenticated addressing between nodes; we layer a couple dozen lines of BPF C on top of that to extend the authentication model to customer address prefixes. So, effectively, we're using WireGuard as an identity. In fact: we do so explicitly for peering connections to other services.

So yeah, it's a model that can work. It's straightforward for us because we have a lot of granular control over what can get addressed where. It might be trickier if your network model is chaotic.

I too am interested in this.

I long for the day where Kubernetes services, virtual machines, dedicated servers and developer machines can all securely talk to eachother in some kind of service mesh, where security and firewalls can be implemented with "tags".

Tailscale seems to be pretty much this, but while it seems great for the dev/user facing side of things (developer machine connectivity), it doesn't seem like it's suited for the service to service communication side? It would be nice to have one unified connectivity solution with identity based security rather than e.g Consul Connect for services, Tailscale / Wireguard for dev machine connectivity, etc.

One of the methods that Cilium (which implements this eBPF-based service mesh idea) uses to implementation authentication between workloads is Wireguard. It does exactly what you describe above.

In addition it can also be used to enforce based on service specific keys/certificates as well.

You'd be adding a whole new layer of what would effectively be dynamic routing. It's doable, but it's not a trivial amount of effort. Especially if you want everything to be transparent and automagic.

There's earlier projects like CJDNS which provide pubkey-addressed networking, but they're limited in usability as they route based on a DHT.

Ziti (Apache) provides bootstrapped* identity based security (and programmable, least privilege overlays).

Disclosure: founder of company which sells SaaS on top of Ziti FOSS.

* https://ziti.dev/blog/bootstrapping-trust-part-5-bootstrappi...

The model I'm describing contains two pieces: 1) Moving away from sidecars to per-node proxies that can be better integrated into the Linux kernel concept of namespacing instead of artificially injecting them with complicated iptables redirection logic at the network level. 2) Providing the HTTP awareness directly with eBPF using eBPF-based protocol parsers. The parser itself is written in eBPF which has a ton of security benefits because it runs in a sandboxed environment.

We are doing both. Aspect 2) is currently done for HTTP visibility and we will be working on connection splicing and HTTP header mutation going forward.

You can! There are downsides though for any sufficiently polyglot organization, which is maintaining all the different client SDK's that need to use that.

Sidecars are often useful for platform-centric teams that would like to have access to help manage something like secrets, mTLS, or traffic shaping in the case of Envoy. The team that's responsible for that just needs to maintain a single sidecar rather than all of the potential SDK's for teams.

Especially if you have specific sidecars that only work on a specific infrastructure, for example if you have a Vault sidecar that deals with secrets for your service over EKS IAM permissions, you suddenly can't start your service without a decent amount of mocking and feature flags. Its nice to not have to burden your client code with all of that.

Also, there is a decent amount of work being done on gRPC to speak XDS which also removes the need for the sidecar [0].

[0] https://istio.io/latest/blog/2021/proxyless-grpc/

Author of linkerd argues that splitting this responsibility will improve stability as you'll have a homogeneous interface (sidecar proxy) over a heterogeneous group of pods. Updating a sidecar-container (or using the same across all applications) is possible, whereas if it's integrated into the application you'll encounter much more barriers and need much wider coordination.

There are good reasons more often than not.

Being able to pick up something generic rather than something language-specific.

Not having to do process supervision (which includes handling monitoring and logs) within your application.

Not making the application lifecycle subservient to needs such as log shipping and request rerouting. People get sig traps wrong suprisingly often.

Like it or not the socket has become the demarcation mechanism we use. Therefore all software ends up deployed as a thing that talks on sockets. Therefore you can't/shouldn't put functionality that belongs on the other end of the socket inside that thing. If you do that it's no longer the kind of thing you wanted (a discrete unit of software that does something). It's now a larger kind of component (software that does something, plus elements of the environment that software runs within). You probably don't want that.

OP here: It's whimsical.com. I really love it.

Then you should interview again but with us.

This is not too different from wpa_supplicant used by several operating for key management for wireless networks. The complicated key negotiation and authentication can remain in user space, the encryption of the negotiated key can be done in the kernel (kTLS) or, when eBPF can control both sides, it can even be done without using TLS but encrypting using a network level encapsulation format to it works for non-TCP as well.

Hint: We are hiring.

The answer to this is simple - TLS will start being terminated at the pods themselves. The frontend load balancer will also terminate TLS - to the public sphere, and then will authenticate it's connection to your backends as well. Kubernetes will provide x509 certificates suitable for service-to-service communications to pods automatically.

The work is still in the early phases, so the exact form this will take has yet to be hammered out, but there's broad agreement that this functionality will be first-class in k8s in the future. If you want to keep running proxies for the other feature they provide, great - They'll be able to use the certificates provided by k8s for identity. If you'd like to know more, come to on of the SIGAUTH Meetings :)

What the proposed architecture allows is to continue using SPIFFE or another certificate management solution to generate and distribute the certificates but use either a per-node proxy or an eBPF implementation to enforce it. Even if the authentication handshake remains in a proxy but data encryption moves to the kernel then that is a massive benefit from an overhead perspective. This already exists and is called kTLS.