Benchmarking BDB, CDB and Tokyo Cabinet on large datasets (opens in new tab)

I agree, plus TC has a ton of parameters that can be tweaked, and the defaults are pretty small. The one that has the most pronouced effect is the bucket size, or the "width" of the hash table. The bigger, the less chance of collisions, which means you have to follow a linked-list to find the exact record. He used 11M keys, so a bnum in the range of 40M would be much quicker.

I benchmarked TC b+tree on a 1TB db with ~350M keys, and it worked great. I would publish the numbers, but I'm embarrassed that they aren't very rigorous.

cdb docs say it has a limit of 4GB, which makes it pretty much worthless for anything I would use it for.

A caveat to TinyCDB in Windows:

Don't forget (In Windows) to specify O_BINARY eg:

  O_RDONLY |O_BINARY

Also the case in Berkeley DB 1.86.

Otherwise it opens a file in ASCII mode, when Windows writes \r\n when it reads in \n... , and dies after just a few records.

The first lines of cdb.c and cdb.h read "/* Public domain */". I would expect DJB to spell this out more explicitly if he releases an updated cdb package. I say "if" because the last release was Feb 2000.

DB up to and including 1.86 are Berkeley license (basically 'do anything with this but include this notice' but without open source requirements), but DB up to and including 1.86 only allow < 2GB databases. :-(

I tried to upgrade it with 64-bit integers, but it's such convoluted C code I can't make head nor tail of it.

Also try buying BDB ($20,000). I emailed them asking for a discount, but Oracle (who owns them now) never replied. :-(

You can just talk to BDB (running in a separate process) over an RPC server. Yes, you have to write that server... but it would be trivial to do, and certainly less costly than licensing BDB.

As someone mentioned up above, the author probably didn't tune the bucket number appropriately.

I am a bit confused by the table. "bdb_btree 14.3/s " means 14.3 records per second? Is that the correct interpretation?