zstd

They have a nice table on that page: https://github.com/facebook/zstd#benchmarks Looking at that table, I think LZ4 is a winner. The compression ratio is not too far, compression speed is slightly faster, decompression speed is significantly faster, the code is much simpler so the compiled binary is smaller, and the project is unrelated to Facebook.

2.97. https://github.com/facebook/zstd/issues/1087#issuecomment-44...

Approximately no one uses the 4-clause BSD (the problematic one). The one Zstd uses has three clauses: https://github.com/facebook/zstd/blob/dev/LICENSE https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_...

zstd still doesn't support seekable compressed disk images, the main advantage that xz has: https://github.com/facebook/zstd/issues/395#issuecomment-535...

Fortunately, zstd seems to be in active development: https://github.com/facebook/zstd As far as I can tell, most of zstd's development is still by Facebook employees, though not all of it. I tend to think zstd has enough traction that development would continue even if FB were to abandon the project.

One reason to use tar without gzip is that you can random access files within the tarball without unpacking it. tar + xz with --block-size also allows this. https://libguestfs.org/nbdkit-tar-filter.1.html https://libguestfs.org/nbdkit-xz-filter.1.html Unfortunately zstd still does not: https://github.com/facebook/zstd/issues/395#issuecomment-535...

For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli, is another modern format with some good features for high compression levels, which also has Content-Encoding support in web browsers. You might also want to play with compression levels (-1 to -11 or more, zstd's --fast=n); they may not map 1:1 to gzip's. One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things that would have been impractical in the 90s but are reasonable with modern hardware. The new algorithms also contain lots of smart ideas and have fine-tuned implementations, but that core differen

Yes, sort of: "Zstd offers a training mode, which can be used to tune the algorithm for a selected type of data. Training Zstandard is achieved by providing it with a few samples (one file per sample). The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression. Using this dictionary, the compression ratio achievable on small data improves dramatically." https://github.com/facebook/zstd#the-case-for-small-data-com...

zstd: https://github.com/facebook/zstd Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)

zstd 1.3.8 and later also has an rsync-friendly mode (via the --rsyncable command-line option). See https://github.com/facebook/zstd/issues/1155

zstd still doesn't have a seekable format as part of the official standard (I wish it did): https://github.com/facebook/zstd/issues/395#issuecomment-535...

Correct, xz is no longer particularly useful, mostly annoying. For read-only long-term filesystem-like archives, use squashfs with whatever compression option is convenient. The cool thing about squashfs is you can efficiently list the contents and extract single files. This is why it's an ultimate best option for long-term filesystem-esque archives. https://en.m.wikipedia.org/wiki/SquashFS For everything else, or if you want (very) fast compression/decompression and/or general high-ratio compression, use .zstd. https://github.com/facebook/zstd I've used this combination professionally to great effect. You're welcome :)