dwarfs

Or pax. https://linux.die.net/man/1/pax Or eStargz. https://github.com/containerd/stargz-snapshotter Or Nydus RAFS. https://github.com/dragonflyoss/nydus Links for your mentioned zpaq and dwarFS https://www.mattmahoney.net/dc/zpaq.html https://github.com/mhx/dwarfs

This seems like the perfect use-case for DwarFS : "DwarFS is a read-only file system with a focus on achieving very high compression ratios in particular for very redundant data." https://github.com/mhx/dwarfs

I wonder if there is any benefit to using something like DwarFS (https://github.com/mhx/dwarfs) for something like this.

https://en.wikipedia.org/wiki/Locality-sensitive_hashing https://en.wikipedia.org/wiki/Nilsimsa_Hash https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md#nils...

https://github.com/mhx/dwarfs?tab=readme-ov-file#with-erofs It's been a while since I did that comparison, so the results could be significantly different now.

It's in the man page[1], you put the file at the filesystem creation: $ mkdwarfs -i /path/dir -o image.dwarfs [1]: https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md

Try this one? https://github.com/mhx/dwarfs It has a ton of comparison with existing tools in the README -- zpaqfranz included -- and it seems to be the best there is.

I haven't had a chance to use it yet, but https://github.com/mhx/dwarfs claims to be times faster than squashfs, to compress much better, and to have full FUSE support.

https://github.com/mhx/dwarfs "DwarFS compression is an order of magnitude better than SquashFS compression, it's 6 times faster to build the file system, it's typically faster to access files on DwarFS and it uses less CPU resources."

I thought DwarFS[0] is better than SquashFS? [0] https://github.com/mhx/dwarfs

I typically don't make use of standard archive formats these days for my own file storage. If I want just pure maximum compression, I'll often use a long-range matcher like FreeArc's srep [1] or lrzip [2] combined with either fast-lzma2 [3] using a p7zip fork [4] for multithreading and fast compression or use mcm [5] or zpaq for max ratio with longer compression time. However, my truly preferred way is using dwarfs [6], which features some really good deduplication and (by default) zstd compression while being mountable. Most of my files are highly compressed and easily accessible without needing to full decompress them. I even made a small script to convert and create AppImages that instead use this [7]. Admittedly, I don't make use of PAR2 or anything of the sort, but I could just do that the traditional way if I so wished. [1]: https:/

Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs I initially built this for having access to 1000+ Perl installations (spanning decades of Perl releases). The compression in this case is not quite as impressive (50 GiB to around 300 MiB), but access times are typically in the millisecond region.

Yes, and I've documented this now: https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...