DwarFS: A fast high compression read-only file system (opens in new tab)

Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.

Nowadays you can have the same problem with Python and Javascript too!

I have about the very same problems as mhx, several hundreds of huge perl versions which are almost the same, taking up enourmous amounts of diskspace. E.g. I had to move most of them from my SSD to a spinning disk. I really want to move them back.

Thanks to mhx I can move them now back to my fast disk. This is also perfect for testers.

nice, wonder how this compare with MongoDB compression of files and objects. Seems like a great foundation for archiving data.

> @OP: Can you please explain why you keep 50 gigs of perl around? :-)

Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release was tested against basically every possible version (and several build flag permutations) of perl that was potentially out in the wild.

Speculating here, but perl has a very rich test library and harnesses for running tests across multiple perls and platforms.

If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations:

http://matrix.cpantesters.org/?dist=Log-Any-Adapter-FileHand...

AppFS provides global file deduplication and also solves the distribution problem, and also you don't need to have all the resources locally

You could use a [WIM image][1]. They can be mounted rw or ro and have file-level deduplication. Microsoft's official tooling is necessary to mount them on Windows as [the only open source implementation I am aware of][2] uses FUSE for mounting.

[1]: https://en.wikipedia.org/wiki/Windows_Imaging_Format [2]: https://wimlib.net/

If you are using windows server, data deduplication[1] is available on non-system volumes to do exactly this.

If you're using a Windows client, there is a way of enabling this, but it's not exactly supported, for a variety of reasons.

[1]https://docs.microsoft.com/en-us/windows-server/storage/data...

It's probably a hack, but you can try "backing up" your files with bup, restic or borg, and mount the resulting snapshot with FUSE

You probably need to de-duplicate before compression, at least for many compression schemes.

s/ask/request/g

Author here :)

I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but to make it better in certain scenarios (huge, highly redundant data) than SquashFS. SquashFS still has the big advantage of being part of the kernel, which makes it a lot more attractive for things like root file systems.

I don't know about you guys, but most of the stuff that takes up space on my drives are:

1) Videos from my DSLR

2) RAW images from my DSLR

3) Various movies / TV series I downloaded

4) Game files (most of which are textures and 3D models)

None of that stuff is really compressible.

You could probably build something easily in nbdkit to do this. (Note this is at the block layer). An advantage of nbdkit is you could write the whole thing in the high-level language of your choice, even a scripting language such as Python, which might make it easier to rapidly explore designs.

Having said that I did try to implement a deduplication layer for nbdkit, but what I found was that it wasn't very effective. It turns out that duplicate data in typical VM filesystems isn't common, and the other parts of the filesystem (block free lists etc) were not sufficiently similar to deduplicate given my somewhat naive approach.

I believe the term of art that applies here is "Hierarchical Storage Management". Along with automatically moving data between high-cost and low-cost storage media, the low-cost storage media for your filesystem of choice for the kind of compressing you described can simply be fast disk on a compressing filesystem.

I believe NT file compression works like this, and before that MSDOS "DriveSpace" ...

Checkout CVMFS

It is not what you describe but it can help.

This is a read-only file system, so it’s able to exploit certain properties of that—- locating similar files next to each other, for example.

Is Btrfs stable yet?

> $ time mksquashfs install perl-install.squashfs -comp zstd -Xcompression-level 22

> Parallel mksquashfs: Using 12 processors

I guess something like borgbackup would be better suited for this.

You could theoretically try to build this with dwarfs, by using overlayfs and then compressing the upper layer again with dwarfs, but that sounds pretty fragile and cumbersome.

Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU resources. However, it'll still need a reasonable amount of memory at run time to cache active, decompressed blocks.

I was thinking the same thing! I'm not sure what it would take to make /rom a FUSE based filesystem, to make it bootable. The current boot process involves the bare kernel mounting Squashfs to find it's init=/etc/preinit & booting from there[1].

Would love some theorycrafting on possible ways to work with DwarFS being a FUSE filesystem.

[1] https://openwrt.org/docs/techref/process.boot

It was DoubleDrive until Microsoft licensed it and relabeled it as DoubleSpace. Stacker was the far more popular drive compression solution until MSDOS 6 was released.

> mkdwarfs crashed with recursive links (1-level, just pointing to itself)

That's odd, it shouldn't crash with links at all, as it doesn't actively follow links. Can you please file a bug if you can reproduce this?

> and when I removed dirs while running mkdwarfs, which were part of of the input path

I guess this is fair, but I'll try to take a look anyway. :-)

> On success, mkdwarfs needed 1 hr, and reduced 219 dirs to a size of 970 MB. Not just source files, but also the build and install object files.

My 500 MB image with the 1100+ perls is just installations, from which I've actually removed libperl.a as I've never needed it and it really bloats the image. I've got a separate image with debug information (everything built with -g in case I need to debug the binaries), so the binaries in the main image are essentially all stripped. If I need to debug, I'll just mount the debug image as well, which contains the source files and the stripped debug data.

> 1 hr is a lot, but just think how long squashfs would have needed.

It might be worth trying a lower compression level, especially if you find that mkdwarfs is CPU bound and not I/O bound.

On success, mkdwarfs needed 1 hr, and reduced 219 dirs to a size of 970 MB. Not just source files, but also the build and install object files.

1 hr is a lot, but just think how long squashfs would have needed. Totally impractical. Thanks mhx

Most ZFS howto's now recommend against dedup on the prolongued memory cost consequences. Yes, you would get some block level compression outcome. But, you enter the cost/benefit hell of balancing CPU and memory at runtime.

My Python shits have pip requirements that easily dump 3-4 gigs in a venv folder. Do that once or twice a month when starting a new project for a couple of years and it gets messy...

This is really cool, I'll give squashfs-tools-ng a try!

> Does the Raspbian image comparison compare XZ compression against SquashFS with Zstd?

That's correct. It's not an exhaustive matrix of comparisons.

> Also, is there any documentation on how the on-disk format for DwarFS and it's packing works which might explain the incredible size difference?

The format as of 0.2.0 is actually quite simple. It's a list of compressed data blocks, followed by a metadata block (and a schema describing the metadata block). The metadata format is implemented by and documented in in [1].

There are probably 3 things that contribute to compression level:

1) Block size. DwarFS can use arbitrary block sizes (artificially limited to powers of two), and uses a much larger block size (16M) by default. SquasFS doesn't seem to be able to go higher than 1M.

2) Ordering files by similarity.

3) Segment deduplication. If segments of files overlap with previously seen data, these segments are referenced instead of written again. The minimum size of these segments can be configured and defaults to 2k. For my primary use case, of the 47.6 GB of input data, 28.2 GB are saved by file-level deduplication, and another 12.4 GB by this segment-level deduplication. So before the "real" compression algorithms actually kick in, there are only 7 GB of data left. As these are ordered by similarity, and stored in rather big blocks, some of the 16M blocks can actually be compressed down to less then 100k.

[1] https://github.com/mhx/dwarfs/blob/main/thrift/metadata.thri...

I just want to say thank you for squashfs-tools-ng. For my usecase I had to patch mksquashfs and your tool fits just right. I'm yet to switch however.

> have you investigated why this might be the case?

Very briefly. It looks like clang has a different strategy breaking up the code (which is mostly C++ templates) into actual functions vs. inlining it, and the hot code ultimately performs fewer function calls with clang than it does with gcc. But this is nowhere near a proper analysis of what's going on. :)

I think it uses FUSE, which is linux specific.

Read-only media, for example. Or in general, stuff that doesn't really change. In my case: https://github.com/mhx/dwarfs#history

Game asset packages - all game assets are read only and need to be compressed and nowadays with SSD's you don't want duplication.

Just to clarify that last statement (and something to think about) with HDD's you want duplicate assets so that you don't cause seeks which are VERY slow on 5400rpm HDD's still found on some/alot of systems.

Have you ever used a CD-ROM or DVD-ROM?

the use case for a read only compressed filesystem is that one..

* can search archived files potentially faster because read access is potentially faster

* fit more data on bootable media

Booting. Arguably all containers, too.

squashfs is widely used in Linux install media.

Node_modules