Just for fun, for more perspective on big data, a human body generates around 1-10M new cells per second, and a cell contains about 10-100GB of information. So a single human is generating 1-100PB/s of data just in the new cells! (Give or take a few OOM)
OTOH the amount of "information" needed to perfectly simulate a cell is probably unbounded. Just a corollary of the fact that we currently don't know how to perfectly simulate reality. Even a single "real" number can take up infinite space.
No storage system can store that data (and most of it is not useful) so they have a series of hardware triggers and buffers that reduce the data down to roughly what modern (general purpose) hardware is capable of handling. They tune the thresholds to match what consumer hardware is capable of.
With regard to supercomputer filesystems: nobody wants to use GPFS. CERN's EOS sustained (theoretical) 3.3TB/sec in Apr 2015, so it's not like they're uncompetitive with the largest supercomputer...
Those of us who don't own our own cross-ocean fiber can't afford to design systems like this.
(I'm paraphrasing an old, old joke.)
We used it at a company I worked for, but it had long-since been deprecated, so I was confused when I saw this Scribe.
https://web.archive.org/web/20120301000000*/http://www.scrib...
The initial stuff seems to not be so related, but the current description of what they do seems much much closer to what Facebook's Scribe does today :)
Naming is hard! :D
The article just focuses on certain areas of the system and doesn't go into the security and privacy parts, that's all.
(I work in Scribe)
On the volume of metadata held by Producers, will there be any significant difference between holding WriteService & LogDevice meta.
Despite that, I find the claims to be underwhelming. So your system can process massive amounts of data by scaling massively horizontally...neat.
(disclaimer: I work in Scribe)
