It could help you replace the TUN with something more cross platform, and possibly with less overhead. You can pass in the hostname using %h, so you can even have virtual DNS.
https://github.com/majek/slirpnetstack/
(btw, gvisor netstack, while not without problems, is likely to be faster than libslirp, see benchmarks https://github.com/rootless-containers/rootlesskit/pull/101#... )
Having complete control of TCP/IP in userland like this, with so little code, is so valuable I feel like there needs to be some special name for the technique.
The whole thing is kind of a vindication for Go's standard library network interface, which I have always hated.
Yes! Userspace TCP/IP is how we implement firewall for Androids (which don't expose iptables on non-root devices but let you setup TUN interfaces via VPN APIs). Right now, we rely on LwIP (wrapped in golang) and it has worked wonderfully well; especially since it is light-weight without any locking-overheads (single-threaded) and that bodes well for battery-powered devices.
> The whole thing is kind of a vindication for Go's standard library network interface, which I have always hated.
The Fuchsia team at Google is re-implementing netstack3 in Rust (and hence you're probably right to call it "gVisor netstack") due to what I presume are performance and efficiency reasons (which is of interest to us because we develop for smartphones). Of course, flyctl doesn't need that, but since you wrote about pulling in heavy dependencies, I am interested in your take on it.
As a non-Android developer, I've been working on a project the last few months that involves running an HTTP server on the device and tunneling out so it can receive requests from the outside world, and the platform feels nerfed at every level from filesystem access to keeping your server from being battery-killed.
I'll note that while all of gVisor's user-mode Linux is in the same Go module, we've actually gone to decent lengths to keep the network stack logically separate from the rest of the user-mode Linux code.
So while go.sum might look a bit frightening, Brad's depaware shows that the extra code you pull in to binaries by using netstack is actually quite minimal: https://github.com/tailscale/tailscale/commit/5aa5db89d6a9a6....
Curious about this. I've generally found Go's net libs to be pretty pleasant. Can you compare/contrast it with others you like better?
Anyways. Wrong about that one! Movin' on!
Many years ago, when we could take always-on desktop PCs more or less for granted, I developed a product that let the user connect back to their home PC from another PC, to stream music from home or grab a file (this was also pre-Dropbox). NAT was already ubiquitous by this point, and Windows XP SP2 (first version with Windows Firewall) came out that year, so I knew it couldn't just make a direct TCP connection to the user's home PC. So I did a stupid relay implementation, where both the client and the home server (that's what we actually called the tray applet on the home PC) made outgoing TCP connections to our central server, which would relay packets back and forth. If I'd had access to a TCP-in-userspace thing like the gvisor network stack, I could have run TCP end-to-end, the way it's meant to be used. It almost makes me want to reimplement that old system using Go and WireGuard, even though the functionality is basically irrelevant in today's world.
ssh dogmatic-potato-342@jump.fly.io
And tunnel connection over wireguard on jump server
Also, it saves moving cables around when I break stuff.
I really enjoy this style of writing from a company.
Regarding the article, it seems like Fly has pulled off some insane networking nonsense, but I don’t know enough about networking yet to understand it. Saving this page for later and gonna get back to the TCP/IP Guide.
Fly is essentially building a Tailscale-esque infrastructure to service one part of their cloud offering. It is indeed insane the amount of heavy-lifting they do to make it all work. They seem like a cross between packetfabric, gitops, docker, and hashicorp but with way less engineers on the team.
Most of the time these implementations are too tightly tied to the rest of the company's infra to be useful standalone. When one of those companies succeeds a common pattern is for engineers to cash out, leave, and build a new startup around one technology from the success story.
I would not be surprised if this is one of the forces that drives the consumer -> infra -> consumer -> infra cycle. A consumer wave leads to inventing lots of interesting but bespoke infra while it is growing like crazy. When it plateaus, folks spin out the interesting infra bits until the next consumer wave (generally larger) starts rising.
We mostly just try to pick the right primitives. And frequently get that wrong. Like that time we wrote our own JS runtime ...
I'm hopeful we'll also see some robust QUIC-based tunneling tools over the next couple years.
It also seems rather obvious to extend WireGuard to run over QUIC in addition to UDP. But the movement on that front has been very limited.
Also, this post prompted me to look closer at Fly.io, and it's leapfrogged to the top of my shortlist for an imminent client "edge proxy" project.
Last year I implemented TCP/IP over AWS Cloudwatch. Tons of "can you believe that actually works?" stuff possible with it:
https://medium.com/clog/tcp-ip-over-amazon-cloudwatch-logs-c...
Just realized this was written by security guru tptacek, nice. What is the contextual meaning of “AFFIANT SAYS NOTHING FURTHER.”?
"That's all, folks"
Normal SSH still works, and is usually going to be what people end up using. You just have to have WireGuard installed and running.
The product feature here is less interesting than how we did it.
(I admit that I haven't looked much into mesh networking / edge servers, so I don't know what the problems are. I always preferred Internet -> Identity Aware Proxy type thing -> mTLS mesh that is useless to humans. And, I don't ssh to stuff much anymore... I have my software collect debugging information and send it to something I can access through a browser or API, and control that software through an API. So everything is editing config files, basically, not SSHing places ;)
WireGuard is dead simple, but setting it up is extra cognitive friction if you've never dealt with it before (or if you're in an environment where you can't create a network interface). Jason Donenfield did some magic with a Google user space networking stack that lets us "hide" the wireguard component. People using our CLI will soon be able to connect to their private network + SSH into a container with one command.
Basically, WireGuard is cool and being able to connect into a wireguard network from a userland program is really helpful for building a straightforward UX.
What is the relationship with micro kernels? Is the feature available separate from the deployment/hosting?
By the way, as an elixir developer Fly.io looks extremely cool. But my (mostly public sector) customers want to hear something similar to the words "AWS" when asked about hosting – so is it running on top of AWS or Azure or GCP? (instances look like they may be GCP, which is fine too).
My first thought was "Wow, can we make this _more_ complicated please?", and then I read the rest of the post.
I hate technology.
Perhaps it's just me, but this is something I would accept as a "hey, I was bored and worked on something on my free time. It's probably broken but nobody cares because it's a toy thing, but it's sooo cool". I wouldn't accept it as " Fly.io OKR 1.3 (2021): SSH and User-mode iP WireGuard"... it's sounds pretty much like a hack.
Wait until I find a reason to put a whole virtual memory manager into `flyctl`. I'll probably knock out a whole bunch of MBOs that way, and gVisor has me covered.
Any Git opening will do, for private cloning.
