undefined | Better HN

0 pointsandersmurphy2mo ago0 comments

Why? Sqlite and LMDB make fantastic use of it. For anyone doing a single writer db it's a no brainer. It does so much for you and it does it very well. All the things you don't have to implement because it does it for you:

- Reading the data from disk

- Concurrency between different threads reading the same data

- Caching and buffer management

- Eviction of pages from memory

- Playing nice with other processes in the machine

Why would you not leverage it? It's such a great fit for scaling reads.

0 comments

cmrdporcupine2mo ago

The strongest argument as far as I can see it is... the problem is you now lose control over all those things. It's a black box with effectively no knobs.

Anyways, read for yourself, Pavlo & Leis get into it in detail, and there's benchmarks:

https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf

https://db.cs.cmu.edu/mmap-cidr2022/

andersmurphyOP2mo ago

What am I missing? The transactional safety problem (the bulk of the paper) is solved simply with a single writer. Which is where you want to be anyway for efficient batching throughput (and isolation).

The other concerns seem to imply there are no other programs running on the same machine as the database. The minute that's not true (is it ever true?). Then OS will do a better job (as seen with LMDB etc).

I think it's telling that the paper focuses on mongoDB not LMDB.

hyc_symas2mo ago

Fun footnote: SQLite only got on board with mmap after I demonstrated how slow their code was without it. I.e., getting a 22x speedup by replacing SQLite's btree code with LMDB https://github.com/LMDB/sqlightning

andersmurphyOP2mo ago

Thank you for beating the mmap drum and LMDB! It's truly an incredible piece of tech.

alexpadula2mo ago

“ It's such a great fit for scaling reads.”

And losing them.

andersmurphyOP2mo ago

How so? LMDB, boltdb/bbolt and sqlite (with mmap) are all rock solid. Just because mongodb used mmap badly does not make it any less valuable.

j / k navigate · click thread line to collapse

0 pointsandersmurphy2mo ago0 comments

- Reading the data from disk

- Concurrency between different threads reading the same data

- Caching and buffer management

- Eviction of pages from memory

- Playing nice with other processes in the machine

Why would you not leverage it? It's such a great fit for scaling reads.

0 comments

cmrdporcupine2mo ago

The strongest argument as far as I can see it is... the problem is you now lose control over all those things. It's a black box with effectively no knobs.

Anyways, read for yourself, Pavlo & Leis get into it in detail, and there's benchmarks:

https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf

https://db.cs.cmu.edu/mmap-cidr2022/

andersmurphyOP2mo ago

I think it's telling that the paper focuses on mongoDB not LMDB.

hyc_symas2mo ago

andersmurphyOP2mo ago

Thank you for beating the mmap drum and LMDB! It's truly an incredible piece of tech.

alexpadula2mo ago

“ It's such a great fit for scaling reads.”

And losing them.

andersmurphyOP2mo ago

How so? LMDB, boltdb/bbolt and sqlite (with mmap) are all rock solid. Just because mongodb used mmap badly does not make it any less valuable.

j / k navigate · click thread line to collapse