undefined | Better HN

0 pointszupa-hu2y ago0 comments

Care to elaborate? How is reading from an append-only file backed by a memory indexed DB slower compared to either 1) a mutated file, or 2) either append-only or mutated raw NVMe disk storage?

I mean, what's the trick NVMe can do to be meaningfully faster?

0 comments

LAC-Tech2y ago

Your views are intriguing and I wish to subscribe your news letter.

But seriously, I've been thinking about an append-only files + memory indexed DB for the past couple of weeks - any prior art or links or papers or anything, lay it on me.

zupa-huOP2y ago

I've been using it in production for 8 years in Boomla. It's closed source though. I haven't found any prior art myself, so just went from first principles. Take a look at the data structure of Git for inspiration. (Merkle tree)

Write speed wasn't my primary motivation though. I wanted a data storage solution that is hard to fuck up. Hard to beat append only in this regard. Plus everything is stored in merkle trees like in Git, so there is the added benefit of data integrity checks. Yes, bit rot is real, and I love to have a mechanism in place to detect and fix those.

j / k navigate · click thread line to collapse