It’s solving a real problem, but not considered stable for production yet.
We have been using SQLite in WAL mode for over half a decade and never witnessed this. Several of our databases can see concurrent access from hundreds of users with transactions in the 1-10 megabyte range, so I find it a bit odd this never came up.
From the link, "if a writer writes to the database while a checkpoint is ongoing [...] it does mean that the wal file may grow indefinitely if the checkpointer never gets a chance to finish without a writer appending to the wal file. There are also circumstances in which long-running readers may prevent a checkpointer from checkpointing the entire wal file - also causing the wal file to grow indefinitely in a busy system."
SQLite runs in a surprising number of places. In an embedded environment disk space may be limited.
I've had to restart services when WAL files had grown to multiple GBs and wouldn't shrink.
The recovery log growing large is explained in [1] - "it does mean that the wal file may grow indefinitely [...] There are also circumstances [...] causing the wal file to grow indefinitely in a busy system."
Not ready for production is [2] - although that is from 2 years ago, the WAL2 branch still isn't merged to trunk [3] which you'd expect if it was ready to go, I think?
[1] https://www.sqlite.org/cgi/src/doc/wal2/doc/wal2.md [2] https://sqlite.org/forum/forumpost/17249fb83a?t=c&unf [3] https://www.sqlite.org/cgi/src/timeline?r=wal2
Reading between the lines a bit, I suspect unbounded growth is still possible with WAL2 - If you have an infinitely long running read transaction, data that modifies the page that’s being read from can never be checkpointed into the main DB.
So setting aside the case of readers that never progress, with WAL1, even if your read transactions were finishing and fast forwarding through commits, it was possible to be in situations where, for a very long time (even potentially forever), at least one reader was behind the most recent write transaction, which meant the wal could never be truncated.
Now with this WAL2 mode I think it should be guaranteed that so long as your read transactions are eventually ending and progressing towards more recent commits, you’ll always eventually find a time to checkpoint and truncate your WAL2 files.
Though you still have to occasionally support arbitrarily large WAL files to an extent because there’s no limit to how big any one write transaction is.
This way an SQLite library which has never heard of a WAL2 file will just use WAL mode instead, instead of potentially being confused by the unheard-of existence of multiple .wal files.
