undefined | Better HN

0 pointsredbad14y ago0 comments

I had temporarily forgotten about master-slave replication. Thanks for the reminder. And I appreciate your broad points. But I still maintain that you're fooling yourself if you think any of those options is a stand-in for Proper™ database-style persistence. By that I mean some contractual assurance in the software that a change written to a data structure is permanent, to standards comparable to the underlying storage system.

If Redis were a database, I would expect a successful HMSET to generally be available in perpetuity, even if the machine was rebooted immediately after I got the "OK".

Append-only command logs don't solve the problem; replaying a complex series of state transitions is not a viable substitute for the storage of the end result of those transitions. It's computationally correct, and a straightforward implementation, but quickly and easily becomes unacceptably inefficient. I hope that's clear enough that I don't need to provide an example.

Replication is a solution for the problem of network or machine instability, assuming a valid Redis use-case. It doesn't address persistence in the sense of databases. In distributed computing, High-Availability is orthogonal to Persistence.

Periodic background flushes to disk in another thread come the closest to solving the persistence problem, but to get database-class QoS you'd need to trigger on every change (or, say, every second). Obviously this is a bad idea and not what the feature is designed for, which circles back to my main point: Redis is not designed to be a database. If it's operating as a memcached replacement in your stack, great. If it's standing in for authoritative, long-term storage of critical data, it's being misused.

0 comments

pjscott14y ago

Let's compare Redis's AOF mode with InnoDB. The way that InnoDB manages to give this guarantee is by flushing its log to disk on every transaction commit. If you're willing to sacrifice some durability for write speed, this can be relaxed. In redis, the closest equivalent to this would be running in AOF mode with a flush on every write.

The difference here is not one of durability, but in how the data is stored on disk. InnoDB keeps the logs small by periodically updating a B-tree with the changes in the logs, after which those changes can safely be removed from the logs. The result of this is strong durability, a reasonably compact on-disk representation, and fairly fast recovery when someone trips over the power cord.

Redis, in AOF mode, logs every command to the log file and (if you specify it in the config file) flushes to disk after every write. The problem is that this file grows without bound: if you leave redis running forever, it will eventually fill up your hard drive, and recovering from a restart will take way too damn long if you have to replay a 1 TB log file. The conventional way of dealing with this is to periodically use the BGWRITEAOF command, which does essentially the same thing as a background data dump: it writes out a new AOF file from the current contents of redis in memory, and deletes the old AOF file. This is roughly equivalent to augmenting the usual periodic-data-dump behavior of redis with periodically-flushed logs, just like a more conventional database.

If there's something I'm missing here, I'd love to hear it.