This should really be upstreamed as an option on the ssh library. Its good to default to sending chaff in untrusted environments, but there are plenty of places where we might as well save the bandwidth
I come from a world (yesteryear) where a computer had 1KB of RAM (ZX80). I've used links with modems rocking 1200 bps (1200 bits per second). I recall US Robotics modems getting to speeds of 56K - well that was mostly a fib worse than MS doing QA these days. Ooh I could chat with some bloke from Novell on Compuserve.
In 1994ish I was asked to look into this fancy new world wide web thing on the internet. I was working at a UK military college as an IT bod, I was 24. I had a Windows 3.1 PC. I telnetted into a local VAX, then onto the X25 PAD. I used JANET to get to somewhere in the US (NIST) and from there to Switzerland to where this www thing started off. I was using telnet and WAIS and Gopher and then I was apparently using something called "www".
I described this www thing as "a bit wank", which shows what a visionary I am!
Yo, 300 baud, checking in.
Do I hear 110?
+++ATH0
For that matter, why does it need to be encrypted at all? What's the threat model?
If there really is a genuine need to encrypt and low latency is critical, consider using a stream cipher mode like AES-CTR to pregenerate keystream at times when the CPU is lightly loaded. Then when you need to encrypt (say) 128 bytes you peel off that many bytes of keystream and encrypt at close to zero cost. Just remember to also MAC the encrypted data, since AES-CTR provides zero integrity protection.
I'm literally working on a web interface I want to use for classic BBS door play... currently working on a DOS era EGA interface, and intend to do similar for PETSCII/Comodore64/128 play as well. I've got a couple rendering bugs to explore for ansis submitted that messed up in the viewer test mode.
https://github.com/bbs-land/webterm-dos-ansi
It's been an opportunity to play with AI dev as well... spent as much time getting the scrollback working how I want as it took on the general rendering.
This feels like a really niche use case for SSH. Exposing this more broadly could lead to set-it-and-forget-it scenarios and ultimately make someone less secure.
https://github.com/openssh/openssh-portable/blob/d7950aca8ea...
Versus...seeing there's a vulnerability, someone adding a one-line change to disable the vulnerable algorithm, compile, image update, test. And a lot less testing because you're not moving to a new version of the language / compiler.
The man has no practical experience in running a production network service, an ego the size of a small moon, and yet was a major contributor to a security protocol now in use by billions of people.
But hey, you can be a handbag designer and end up head of design at Apple soooooooo
Sure, if it's an internal tool you can recompile both ends and force a universal update. But anything else and you need to stay compatible with clients and anytime you allow negotiation of the cryptosuit you open yourself up to quite a few subtle attacks. Not saying that choice about go is clearly a good one but i don't think it's obviously wrong.
The proper fix would be adding option server-side to signal client it's not needed and have client side have option to accept or warn about that
Another good trick for debugging ssh's exact behavior is patching in "None" cipher support for your test environment. It's about the same work as trying to set up a proxy but lets you see the raw content of the packets like it was telnet.
For terminal games where security does not matter but performance and scale does, just offering telnet in the first place can also be worth consideration.
Also, port 21 is often blocked.
I've used Claude a bit and it never speaks to me like that either, "Holy Cow!" etc. It sounds more annoying than interacting with real people. Perhaps AIs are good at sensing personalities from input text and doesn't act this way with my terse prompts..
I've used Claude for debugging system behavior, and I kind of agree with the author. While Claude isn't always directly helpful (hallucinations remain, or at least outdated information), it helps me 1) spell out my understanding of the system (see [1]) and 2) help me keep momentum by supplying tasks.
I use the Other Voices for that. I can't entirely turn them off, I might as well make use of them!
Also, always reminds me of Kermit singing "...you make bath time so much fun!..."
The comment about Claude being pumped was a joke.
2020s: ha! with some advanced probabilistic models, we may be able to deduce something about what is being typed behind one of our layers of encryption, let's sent 100 packets per keystroke to mitigate that
https://www.brendangregg.com/sshanalysis.html
The 2023 patch should finally fix that 2004 issue.
Time flies
Disabling TCP_NODELAY would also reduce number of packets + be portable & simpler to implement - but would incur a latency penalty.
For people who don't feel like googling it:
1. You TCP_CORK a socket
2. You put data into it and the kernel buffers it
3. If you uncork the socket, or if the buffer hits MSS, the kernel sends the packet
Basically, the kernel waits until it has a full packet worth of data, or until you say you don't have any more data to send, and then it sends. Sort of an extreme TCP_YESDELAY.
See https://catonmat.net/tcp-cork for where I learned it all from.
I am aware of TCP_NODELAY (funny enough I recently posted about TCP_NODELAY to HN[1] when I was thinking about it for the same game that I wrote about here). But I think the latency hit from disabling it just doesn't work for me.
I got a kick out of this comment [0]. "BenjiWiebe" made a comment about the SSH packets you stumbled across in that thread. Obviously making the connection between what you were seeing in your game and this random off-hand comment would be insane (if you had seen the comment at all), but I got a smile out of it.
"hello world" fits in a single TCP packet, but the kernel might end up sending one packet containing "hello" and another packet containing " world". It is completely opaque to userspace.
TCP_CORK lets userspace decide when packets are dispatched. You get to control whether "hello world" is sent across 1 packet or 11 packets.
1) I'm pretty much never typing secrets into an SSH tunnel; these days if there's a secret I need to transmit over SSH I'm going to be copying and pasting it, which will not reveal info from keyboard timing. (Or rsync'ing a file, which ditto).
2) I'm not in a high-security environment where nation-states have an interest in sniffing my keystrokes.
3) I often open SSH connections to servers in other continents. Those underwater cables have massive bandwidth, but they're also in constant use by thousands upon thousands of people. So anything I can do to reduce my bandwidth by 100x is probably worth doing.
Any reason you can think of why I should not be setting ObscureKeystrokeTiming=no in my ~/.ssh/config?
(1) This sounds brittle. Are you really going to have a good mental model about what's secret when using ssh and reliably refrain from typing those things? Seems to kinda defeat the idea of securing the channel. Also, as a collection your activities might be more confidential to you than single inputs, or correlated with your other activities outside ssh, etc - it's hard to keep a mental model of this as well. Aka optimism is not a form of security.
(2) There isn't a reason to think this is a difficult attack that only a powerful adversary could mount. Seems like a college lab level thing to me. And very amenable to AI help as well. Also here optimism is not a form of security. It's a 25 year old attack[1] so there's a lot of existing research[2] around.
(3) Saving 100x bandwidth on single keystrokes on an internet dominated by video traffic just because it's 100x doesn't make sense. Also it's good to cultivate a mindset that steers away from trading off security in favour of trivial resource savings.
[1] https://www.usenix.org/conference/10th-usenix-security-sympo... (probably older stuff exists outside open literature)
[2] eg https://crzphil.github.io/posts/ssh-obfuscation-bypass/
Not sure whether the obfuscation is fully synchronous, i.e waiting for the server response before continuing. That would really kill it. Working with LTS distros I don't think I have seen it in practice yet. Need to try something modern on my next trip abroad.
That said, plenty of people disable the most useful security features of SSH, like verifying host key signatures, with no ill-effects (as far as they know). For the majority of users, using Telnet and unencrypted HTTP would make no difference, as nobody's trying to hack them, and who really cares about privacy anyway?
Did you know SSH has long-standing performance limitations due to its design that need patches to eliminate? It was never intended to be a high-performance tool. If you want really high performance, use Telnet. If you want real security, use SSH with all strong security options enabled plus a server using ContainerSSH with the OAuth2 plugin (SSH's keys are static, which can be captured and reused, which is bad). If you don't care either way, use SSH with the defaults.
One common secret that goes through a tty ssh connection is a sudo password. You are probably typing sudo command so without obfuscation the attacker can find out the sudo keystrokes, the command keystrokes and then the encrypted bytes of the password. They don't have the timing data to decode them as easily as the previous parts but if they record enough traffic they might be able to decrypt the password. But maybe they won't, because the ssh session key is probably different each time. Furthermore I don't know how many times they should capture your encrypted password to be able to decrypt it. Maybe it's unfeasible.
Anyway, in case of the sudo password, if the attacker gets it what would happen? The attacker is hopefully not able to get a shell into the server. If they do they have different ways to get root privileges.
By the way, I also copy and paste secrets from either the password manager or the clipboard, because nobody remembers long random strings. The only exceptions are the passwords of a few accounts.
Now that's solving the problem the wrong way. If you really want that, send all typed characters at 50ms intervals, to bound the timing resolution.
Wouldn't this just change the packet interval from 20ms to 50ms? Or did you mean a constant stream of packets at 50ms intervals, nonstop?
I think the idea behind the current implementation is that the keystrokes are batched in 20ms intervals, with the optimization that a sufficiently long silence stops the chaff stream, so the keystroke timing is obfucated with an increased error bar of 20ms multiplied by number of chaff packets.
So a clock doesn't solve the problem. The amount of data sent on each clock pulse also tells you something about what was sent.
The Chaff packets already fire on a timer. They inject random extra fake keystrokes so you can't tell how many keystrokes were actually made. The only other way I can think of to solve that is by using a step function: Send one larger packet (fragmented or the same number of individual packets) on each clock pulse if the actual data is less than some N where N is the maximum keystrokes ever recorded with some margin. Effectively almost every clock pulse will be one packet (or set of packets) of identical size. Of course if you do that then you'll end up consuming more data over time than sending random amounts of packets.
> I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark.
For some people who are used to using Wireshark and who know what to look for, probably yes. For the vast majority of even technical people, probably not.
In my case, I did a packet capture of a single keystroke using tcpdump and imported it into Wireshark and I get just over 200 'Client: encrypted packet' and 'Server: encrypted packet' entries. Nothing useful there at all. If I tcpdump the entire SSH connection setup from scratch I get just as much useful information - nothing - but, oddly, fewer packets than my one keystroke triggered.
So yeah, I dislike LLMs entirely and dislike the reliance on LLMs that we see today, but in this case the author learned a lot of interesting stuff and shared it with us, whereas without LLMs he might have just shrugged and moved on.
Try debugging that shit. Thats right, debugging interfaces aren't safe, by some wellakshually security goon.
You want a real fun one to debug, is a SAML login to a webapp, with internal Oauth passthrough between multiple servers. Sure, I can decrypt client-server stuff with tools, but server-server is damn near impossible. The tools that work break SSL, and invalidate validation of the ssl.
Yes, Esri products suck. Bad.
Seems because dumping the session keys is not at all a common thing. It's just a matter of effort though - if someone put in the time to improve the SSH story for dissectors, most of the groundwork is there.
The author didn't, and used a general tool to their aid - why is that unfortunate?
My thinking was:
* Yes, I clearly know what tcpdump is / how to capture network traffic
* It has been several years since I have looked at a pcap
* I don't have wireshark installed on this computer
* I've done the thing where you decrypt TLS with wireshark exactly once, years ago, and I found it frustrating for reasons I can't remember[1]. Wasn't sure if I could do this with ssh
* When I started investigating this, I didn't remotely think that ssh was the root cause. I thought it was a quirk of my game
* I *did* make a client that printed out all the data it was receiving, but it was useless because it was operating at the wrong layer (e.g. it connected over SSH and logged the bytes SSH handed it)
* I'm experimenting with Claude Code a lot because it has a lot of hype and I would like to form an opinion
* Looking up flags is annoying
* Being able to tell an agent "look at this pcap and tell me what you see" is *cool*
And the big leap here was realizing that this was my SSH client and not a quirk of my game. The time at which I would have read the SSH spec was after I captured traffic from a regular SSH session and observed the same pattern; before that I was thinking about the problem wrong.
I don't think that this is unfortunate. In fact, I think I got what I wanted here (a better sense of Claude Code's strengths and weaknesses). You're right that an alternative approach would have taught me different things, and that's a worthy goal too.
[1] I suspect this is because I was doing it for an old job and I had to figure out how to run some application with keys I controlled? It would have been easier here. I don't remember.
This looks like an actual productivity boost with AI.
I'm still waiting for a systems engineering tool that can log every layer, and handle SSL the whole pipe wide.
Im covering everything from strafe and ltrace on the machine, file reads, IO profiling, bandwidth profiling. Like, the whole thing, from beginning to end.
Theres no tool that does that.
Hell, I can't even see good network traces within a single Linux app. The closest you'll find is https://github.com/mozillazg/ptcpdump
But especially with Firefox, good luck.
For example, "nc" (netcat) is pre-installed on all platforms where ssh is.
There are two issues with it:
- a primary is not a totality: if "security is the #1 consideration for SSH", that implies there's a #2, maybe even a #3 and so on consideration. So the question that follows becomes tautological: "but if the author doesn't need security, why use ssh?" -> surely for one or more of the #2, #3, etc. considerations, right?
- overabstraction (*): you ended up strawmanning the author. What they had issue with was keystroke timing obfuscation, which is a privacy feature. Timing attacks are (in part) a privacy concern, and privacy is a security concern, yes, but security is not just privacy concern, and privacy concerns are not just about timing attacks; these groups are not equal. For example, they might very well want the transmitted keypresses themselves to remain confidential, or they might very well want to retain cryptographic assurance of their integrity. These are security features they can continue to utilize by sticking with SSH.
All of this is to say, it's not even necessarily them using SSH for a hypothetical #2 or #3 (...etc...) reason, but likely because they still very much want to make use of large chunks of #1, which disabling keypress obfuscation does not actually rid SSH of, only at most weakens it in ways they clearly seem to be okay with.
(*) although if I zoom out enough, this is once again just "a primary is not a totality", just implicitly
This is technically incorrect, because Windows now includes SSH too!
Found your problem.
But it is an interesting world where you can casually burrow into a crypto library and disable important security features more easily than selecting the right network layer solution.
The problems you run into when doing things you shouldn't do are often really fun.
[1] https://news.ycombinator.com/item?id=42342382
You should feel free to explore / abuse all options :)
However, there are existing libraries for exactly this use case - see https://github.com/ValveSoftware/GameNetworkingSockets
I guess QUIC libraries would also work.
running without congestion control means that you avoid slowstart. but at a certain rate you run into poorly defined 'fairness' issues where you can easily negatively impact other flows. past that point, you can actually self-interfere and cause excessive losses for yourself.
quic uses congestion control, but uses latency estimates and variance as a signal to back off. it still imposes an ordering on a per-stream basis. so it might not be ideal either.
sctp has a mode which supports reliable and unordered, which might be something to consider
so really - if you care about latency and have a different reliability model, its worth unpacking all these considerations and using them to select your transport layer or even consider writing a minimal one yourself
Speaking of smoking guns, anybody else reckon Claude overuses that term a lot? Seems anytime I give it some debugging question, it'll claim some random thing like a version number or whatever, is a "smoking gun"
Grok, ChatGPT, and Claude all have these tics, and even the pro versions will use their signature phrases multiple times in an answer. I have to wonder if it's deliberate, to make detecting AI easier?
It's nauseating.
Considering what these LLMs bring to the table, I think a little tolerance for their cringe phrases is in order.
Maybe it has something to do with your profile/memories?
Oh shoot! A shooting.
So the TL;DR of this post is: don't change this setting unless you know what you're doing.
Granted... it would increase the cost (since you're adding reverse proxies) but it would be a quick way to get acceptable latency, rudimentary DDoS protection, and you could try different connection options independent of the main app's logic.
It would be hard to estimate how much latency you're adding with a SSH2 reverse proxy in this case, but it's probably lower than one might think.
The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.] But I loved this write-up. It was readable and explained the problem the OP was encountering and proposed solutions well.
I've been thinking about some stuff like this! Not being able to put my game behind Cloudflare[1] is a bummer. Substantial architectural overhead though.
> The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.]
I hear you, but FWIW the patch I was reverting was trivial (and it's also in the go crypto library, which is pretty easy to read). It's a couple-of-line change[2], and Claude did almost exactly what I would have done (I was tired and would have forgotten to shrink the handshake payload).
[1] This isn't strictly true, Cloudflare spectrum exists, but its pricing is an insane $1/GB last I checked.
[2] https://cs.opensource.google/go/x/crypto/+/833695f0a57b30373...
But... before you think I'm trying to be negative... good on you. I wish you well. Getting crypto/security code into open source projects can be a slog as people frequently come out of the woodwork, so don't get discouraged.
And the more I think about this... there's plenty of examples out there about doing HTTP based reverse proxying, but essentially zero for SSH proxying, so if you do that, it would make a great blog post.
Also I was unfamiliar with SSH being vulnerable in the past to keystroke timing!
2023 discussion about it here.
> I am working on a high-performance game that runs over ssh. The TUI for the game is created in bubbletea 1 and sent over ssh via wish.
> The game is played in an 80x60 window that I update 10 times a second. I’m targeting at least 2,000 concurrent players, which means updating ~100 million cells a second. I care about performance.
High performance with ssh and wish? For sure not. Rather use UDP over secure sockets. Or just normal sockets. Even Claude would come up with much faster code than the ssh/wish nonsense. Or mosh, but this also too complicated.
So one thing I only recently figured out is that using ChatGPT via the web browser chat is massively different from using OpenAI's code-focused Codex model / interface. Once I switched to using Codex (via the VS Code extension + my own ChatGPT subscription) the quality of answers I got improved massively.
So if you're trying to use LLM to help with debug, make sure you're using the right model!! There are apparently massive differences between models of the same generation from the same company
When making this statement, are you taking into account that SSH encrypts the traffic by default?
And in this situation, the amount of encrypted payload in each packet is 36 bytes which is ~40x less than a full packet of ~1500 bytes. You would almost surely hit packet per second limits before you hit payload throughput limits at these small sizes.
Encryption is slow when compared to data throughput you can get with a properly designed transport stack, but that is because it is in comparison to 100 Gbps per core even with no hardware offload. Anything less than ~10 Gbps/1 million packets per second (ignoring other bottlenecks, so only the software transport is the limit) is not merely unoptimized, it is pessimized.
Your assumptions are way off
Step one, run https://www.psc.edu/hpn-ssh-home/introduction/ instead Step two, tune TCP/IP stack Step... much later: write your own "crypto". (I'm using quotes because, before someone points out the obvious, packets-per-keystroke isn't, itself, a cryptographic algorithm, but because it's being done to protect connections from being decrypted/etc, mess with it at your own peril.)
I did add a trackpad monitor though. It shows my raw MacBook trackpad data.
Why not just add random "jitter" to the keystroke packets, but keeping just the 1 actual packet?
> And they’re sent to servers that advertise the availability of the [email protected] extension. What if we just…don’t advertise [email protected]?
The extension is "ping@openssh.com." It shows up in the blog reliably for me across several browsers and devices.
Switching to telnet instead of SSH might be an option.
By 'ssh', you mean 'ssh' (library/program + protocol + encryption + decryption) on top of TCP/IP, on top of the Internet, right?
OK, I'm not against it... but you do understand that there are all kinds of ways for that to slow things down, right?
Your issues may (or may not!) include such things as:
o Nagle's algorithm AKA buffering AKA packets not being sent until N bytes (where N > 1) ready to send, as other posters have suggested;
o Slower encryption/decryption on older hardware (if users with older hardware is a target market, and if the added loss in speed makes an impact in gameplay, depending on the game, this may or may not be the case...)
o The fact that TCP/IP (as opposed to UDP / Datagrams / "Raw" sockets) imposes a connection-oriented abstraction, requiring additional round trips of ACK ("I got the packet") RESEND ("I didn't get the packet") on top of the connectionless architecture that is the Internet (https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...), which adds additonal latency, so, for example if a rural user in Australia experiences a 350ms delay for a raw packet to get to a U.S. server (or vice versa), then TCP/IP might make this 700ms or more, depending on the quality of the connection!
o The speed of the game limited to both the bandwidth and latency of the slowest user (if a multi-player game, and if the game must not update until that user "moves"... again, game architecture will determine this, and it wouldn't be applicable to all games...)
Now, you could use UDP, as other posters have suggested, but then you must manually manage connections and encryption...
That may be the right choice for some types programmers, some types of games/applications -- but equal-and-oppositely it may be the wrong choice for others...
Anyway, wishing you well with your game development!
I haven't used SSH (at least, not in a debug capacity), so I'm not sure what SSH debugging options exist -- but it would be nice if SSH had a full logging debug mode, which would explain exactly WHY it chose to send any given packet that it did along with related helpful information, such as latency/time/other metrics, etc., if it doesn't have this/these feature(s) already...
If you get clever and write a client to ensure sensitive data like passwords or email are sent in a burst you could just use an encryption library just for that data instead.
I assume this is done for novelty. There is also Terminal coffee which is a coffee company that takes orders for delivery over SSH.
Though I would suggest to make mosh available, too. Many nethack servers are available via mosh and ssh. (And in an earlier age, telnet.)
And with good reason. This CVE is from yesterday:
https://nvd.nist.gov/vuln/detail/CVE-2026-24061
> telnetd in GNU Inetutils through 2.7 allows remote authentication bypass via a "-f root" value for the USER environment variable.
please never do that (in production)
if anyone half way serious tries they _will_ be able to break you encryption end find what you typed
this isn't a hypothetical niche case obfuscation mechanism, it's a people broke SSH then a fix was found case. I don't even know why you can disable it tbh.
[1] https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf
I'm baffled about this "security feature". Besides from this only being relevant to timing keystrokes during the SSH session, not while typing the SSH password, I really don't understand how can someone eavesdrop on this? They'd have to have access to the client or server shell (root?) in order to be able to get the keystrokes typing speed. I've also never heard of keystroke typing speed hacking/guessing keystrokes. The odds are very low IMO to get that right.
I'd be much more scared of someone literally watching me type on my computer, where you can see/record the keys being pressed.
- you are listening to an SSH session between devices
- and you know what protocol is being talked over the connection (i.e. what they are talking about)
- and the protocol is reasonably predictable
then you gain enough information about the plaintext to start extracting information about the cipher and keys.
It's a non-trivial attack by all means but it's totally feasible. Especially if there's some amount of observable state about the participants being leaked by a third party source (i.e. other services hosted by the participants involved in the same protocol).
So the "real" keystrokes are 100% the same but the fake ones which are never seen except as network packets are what is randomized.
It's actually really clever.
One thing you notice if you have ADSL is that some services are built as if slower connections matter and others are not. Like Google's voice and audio chat services work poorly but most of the others work well. Uploading images to Mastodon, Bluesky, Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for Tumblr you have to try it twice. I don't what they are doing wrong but they are doing something wrong and not finding out what they're doing wrong because they're not testing and they're not listening to users.
Nobody consulted me about their decision not to run fiber by my house. If some committee decides to make ssh bloated they are, together with the others, conspiring to steal my livelihood and I think it would be fair for me to sue them for the $50k it would take to run that fiber myself.
It's OK if you work for Google where there is limitless dark fiber but what about people in African countries?
It's the typical corporate attitude where latency never matters: Adobe thinks it is totally normal that it takes 1-5s for a keystroke to appear when you are typing into Dreamweaver.
But you cannot just sue a company because their network connected software doesn't work well on slow networks. Let alone a project like OpenSSH. It would be like me suing a game studio because my PC doesn't meet their listed minimum requirements to play the game.
A better analogy is a bank redlining neighborhoods. The cost to run fiber to difficult rural locations pays itself easily if you look at a 25-year time span and is an order of magnitude less than building a new housing unit on the West Coast.
If you want a “1990s” mode, add it yourself or pay some to do it for you.
This is funny to me, because ADSL used to be the fast thing, as opposed to dialup modems.
I mean, for modern version of Openssh it's not exactly wrong. The failure was to tell you why that is the normal behavior.
Vibe coders man...
I could vibecode an SSH zmq daemon in an afternoon.
WAT. Please no.