Ah, I see, it is a data ingress problem not a UI/UX/visualization problem and you are in that awkward gap between RAM and disk bandwidths. So, you want a scheme to reduce ingress bandwidth which would allow you to store more data since you can not offload it anywhere else. From this perspective you would want a log optimized for useful entropy per unit storage which you can translate into a different data structure offline later on that is optimized for UI/UX. The most likely scenarios for that would require a very low level integration with the logging infrastructure itself and be either, as you say, a lossless streaming compression or some kind of lossy aggregation/compression, probably domain-specific, that retains the important data. The key problem here being creating an algorithm fast enough to operate at those data rates given that you get at most a few clocks per byte and no way of pushing the data to more parallel compute otherwise you would be able to just push it into storage.

https://github.com/google/snappy is ~250 MB/second per core and you can compress in chunks of whatever size is convenient for your GUI to uncompress and render.

> too big for RAM

64gb ram sticks are about $700 AUD, which brings the problem into range via RAM.

> coming in too fast to write to disk

Good PCIe4 NVMe drives are pretty fast these days. Samsung 980 PRO is under $300 for 1tb, and claims it can write 5gb/sec.

Even if you only get 2 gb/sec, that brings your RAM requirements down to 90gb, which is not an exotic configuration for a high-performance desktop.

I'm eternally astonished by how far hardware has come. Of course, preprocessing the stream on a cheap FPGA would be more elegant, but it's pretty amazing that consumer hardware could do it at all.

My current solution for anyone reading is "capture just 0.5 seconds of trace, and try and time that 0.5 seconds to contain whatever it is I want to see"

blosc comes to mind.