It’s a mindbender of a paradigm-shift for how to think about local recovery actions in the context of the global consensus protocol!
> Disks and flash devices exhibit a subtle and complex failure model: a few blocks of data could become inaccessible or be silently corrupted [8, 9, 32, 59]. Although such storage faults are rare compared to whole-machine failures, in large-scale distributed systems, even rare failures become prevalent [60, 62]. Thus, it is critical to reliably detect and recover from storage faults.
It's not true in general that a node in a distributed system binds its persisted state to a local disk, or flash device, or any specific implementation of any specific kind of storage system. The storage layer is the responsibility of the node to manage, and irrelevant to the wider distributed system in which the node is a participant. Any of those kinds of storage faults need to be accommodated by the node that utilizes those storage layers, but their specific details don't need to be communicated beyond the specific node where they apply. And it's not at all critical for those nodes to detect and recover from any faults in their storage layer; those faults can easily be communicated to the broader distributed system, which necessarily must be able to handle node-specific failures like those without breaking everything down!
> how to think about local recovery actions in the context of the global consensus protocol!
Local recovery actions, or any other kinds of node-specific details, have no relevance or influence on the information communicated thru the the global consensus protocol. A node can persist its state to a local disk, or to RAM, or to an S3 object, or anything else, and none of these details matter at all to the details that node communicates to other nodes in its cluster.
tl;dr: nodes don't necessarily persist state to local disk
In fact, an early version of TigerBeetle implemented one of these from Cowling and Liskov’s VSR’12 paper.
However, since then, we invested in TigerBeetle’s stable storage, for reasons which I won’t go into further here.
If you are curious to learn more about all the stable storage techniques in TigerBeetle in particular (again, this is not trying to suggest that VSR can’t also run without stable storage, or to deny other techniques such as object storage or tiering!), but if these things are interesting for you, and if you do want to learn more, then I’d recommend diving into “Durability and the Art of Consensus”.
