A bit light on technical details but very fun, very exciting. Kind of sad that such amazing work is no longer quite so public, is no longer something that say Intel is going to talk up in endless details with a product launch. A huge amount of the work & innovation here is extremely specific, extremely private- all this Elastic Fabric Adapter related stuff is advanced systems engineering, close integration of systems, that's Amazon's & Amazon's alone.
Anyhow. This article pairs very well with the "Scaling Kafka at Honeycomb"[1], which I found to be a delightful read on adapting & evolving a big huge workload to ever-improving AWS hardware.
[1] https://www.honeycomb.io/blog/scaling-kafka-observability-pi... https://news.ycombinator.com/item?id=29396319 (38 minutes ago, 13 points)
I can't see a reason to buy or use this product.
(hi Jeff! Hope you're well :D)
* What does the Nitro accelerator look like to the host? . Does the Nitro accelerator present as NVMe devices to the OS host, or is there a more custom thing it presents as? Does the Nitro accelerator use SR-IOV to or something else to present as many different PCIe adapters, per-drive PCIe, or a single PCIe device, or no PCIe devices at all, something else entirely (and if so what)? Are there custom virt-io drivers powering the VMs? How much change has gone into these interfaces in the newest iterations, or have these interface channels remained stable?
* What is the over the wire communication? Related to the above; ultimately the VM's see NVMe, & how far down the stack/across the network does that go? Is what's on the wire NVMe based, or something else; is it custom? What trade-offs were there, what protocols inspired the teams? Originally at launch it seemed like there was a custom remote protocol[1]; has that stayed? What drove the protocol evolution/change over time? What's new & changed?
* What do the storage arrays look like; are they also PCs based? Or do the flash arrays connect via accelerators too? Are these FPGA-based or hard silicon? Are there standard flash controllers in use, or is this custom? How many channels of flash will one accelerator have connected to it? How much has the storage array architecture changed since Nitro was first introduced? Do latest gen nitro & older EBS storages have the same implementation or are newer EBS storages evolving more freely now?
* On a PC, an SSD is really an abstraction hiding dozens of flash channels. There have been efforts like Open Channel SSDs and now zoned namespaces to give the PCs more direct access to the individual channels. Does the Nitro accelerator connect to a single "endpoint" per EBS, or is the accelerator fanning out, connecting to multiple endpoints or multiple channels, doing some interleaving itself?
* What are some of the flash-translation optimizations & wins that the team/teams have found?
And simply: * How on earth can hosts have so much networking/nitro throughput available to them?! It feels like there's got to be multiple 400Gbit connections going to hosts today. And all connected via Nitro accelerators?
It's just incredibly exciting stuff, there's so much super interesting work going on, & I am so full of questions! I was a huge fan of the SeaMicro accelerators of yore, an early integrated network-attached device accelerator. Getting to work at such scale, build such high performance well integrated systems seems like it has so so many interesting fascinating subproblems to it.
You speak my mind: https://news.ycombinator.com/item?id=19162376 (from 3yrs ago)
For example, why not have a file/object/whatever storage service; and a price matrix that lets you select key metrics like latency, throughput, and variability of either?
I don’t particularly care if my ultra fast ultra low latency is derived from SSDs, spinning rust, RAM, l2 cache, or acoustic ripples. But I’m not super in tune with cloud services to begin with.
This custom SSD hardware family is what powers the EBS (cloud SAN) service, which allows you to pay for the performance level you need, where they give it both higher absolute performance and [now] better worst-case latency.
This announcement is saying that you can now get your own instances with the same performance characteristics for situations where you need better performance than a SAN can deliver and/or the robustness benefits of using per-node storage rather than a separate networked service.
The other part of this announcement is the implicit message it sends about the competition: they're telling everyone that their storage performance is more consistent than their competitors and increasing the number of areas where they can say they have an option which a competitor does not. Noting that this was driven by the EBS storage team is also a reminder that they have more people working on lower-level infrastructure problems than you likely do.
They also have things like Lightsail if you don't care about the details and just want the packaged solution.
Comparing to life before the cloud, you would have to choose the vendor, the size of your drives, how many drive, then go through the procurement process, months in advance.
Tl;dr: They now have custom SSD firmware that avoid latency spikes.
Especially in the "Up to X per second" networking instances, which is basically all of them except the huge ones.
The activation of throttles is NOT well exposed in metrics, nor is bursting amount or detecting if bursting is occurring.
It is all somewhat shady IMO, with AWS trying to hide problems with their platform, or hide that you're getting charged in lots of sneaky ways.
[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-opti...
Assuming this means something similar to QLDB, did they put a centralized blockchain in the firmware? Pretty cool.
I think we can probably think more along the lines of the postgres wal (write ahead log) and _journaling_ file systems here.
