undefined | Better HN

0 pointsagentultra3y ago0 comments

How many applications need infinitely scalable distributed object storage?

I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data.

But those were outliers. All of the large and small enterprises outside of that could fit all of their “big data” in the memory of a single blade server and still have plenty to spare. You can get machines these days with many TB’s of RAM.

0 comments

boltzmann-brain3y ago

You can hardly make quantitative statements about how many applications need this or that. It's not like we're in the 60s and IBM can tell that they'll need to make exactly seven of a certain computer.

I can tell you that massive amounts of storage are used by eg Mercedes Benz during simulations they need when developing cars. Back when I was reading about this - maybe a decade ago - they were using mind boggling amounts of storage. Still all before SSDs. You can assume all other world-class engineering firms in every kind of difficult engineering that requires physical robustness or safety have a similar need. But Mercedes Benz at least used an on-premises solution, simply because storing stuff on AWS would have created latency and throughput issues.

agentultraOP3y ago

That does seem to be the common trend. If you generate that much data you tend to host on premises because the network becomes the bottleneck. It becomes important to place compute as close to the data as possible.

treffer3y ago

This is my favorite joke.

Is the issue big data or small machine?

Memory size of the largest AWS/GCP/... instances is a good indicator for small machine.

It doesn't mean I would go for the large machine. It just means I won't subscribe to "big data" as a reason to do X.

And it forces me to recalibrate this boundary regularly. Looks like double digit TB is currently the memory boundary for renting. I was still on the single digit TB train.

j / k navigate · click thread line to collapse