zstd

In general, xz beats zstd in compression ratio, as xz is very committed to providing the strongest compression, at the expense of speed, while zstd provides a range of compression ratio vs speed tradeoffs [0]. At the lower levels, zstd isn't approaching xz's compression level, but it's doing it much much faster. Additionally, zstd generally massively outperforms xz in decompression speed $ time xz linux-4.14-rc6.tar real 4m26.009s user 4m24.828s sys 0m0.724s $ wc -c linux-4.14-rc6.tar.xz 103705148 linux-4.14-rc6.tar.xz $ time zstd --ultra -20 linux-4.14-rc6.tar linux-4.14-rc6.tar : 12.81% (824350720 => 105564246 bytes, linux-4.14-rc6.tar.zst) real 4m34.129s user 4m33.484s sys 0m0.432s $ time cat linux-4.14-rc6.tar.xz | xz -d > out1

lrzip is a preprocessor that finds matches in the distant past that the backend compressor (xz) couldn't normally find. zstd has a new long range matcher mode inspired by the ideas behind rzip/lrzip with some extra tricks. It produces data in the standard zstd format, so can be decompressed the the normal zstd decompressor. There is a short article about it in the latest release notes https://github.com/facebook/zstd/releases/tag/v1.3.2

They dual licensed under BSD and GPLv2: https://github.com/facebook/zstd/issues/335#issuecomment-323... I expect they did a thorough analysis to work out if any of it was either already patented, or patentable. That doesn't mean they don't have patents though.

Interestingly, Facebook did remove its PATENTS clause on another project they have recently - zstd - https://github.com/facebook/zstd/pull/801

zstd uses ANS [1][2] to encode its (literal length, match length, offset) triples. [1] https://github.com/Cyan4973/FiniteStateEntropy [2] https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi...

>> Welcome to the over-reaction party Hmmm.. in that case the people who actually need this advice even more would be the maintainers of ZSTD who seem to have hastily changed the license after seeing the backlash from the Apache policy change. Care to stand by this opinion and leave your comment on the thread which discusses the license change over on GitHub? https://github.com/facebook/zstd/issues/775

I am very happy to report that ZSTD removed PATENTS clause: https://github.com/facebook/zstd/pull/801

I was wondering about a similar issue for zstd compression library. It has a similar BSD+Patentsfile thing. There is an issue with a related discussion about it going for more than a year: https://github.com/facebook/zstd/issues/335 Last update is interesting. Someone did a search and looked for any patents FB filed for and couldn't find any in last year. So somehow based on that they decided things are "OK". To quote: --- US allows to patent up to a year after the publication, allowing to conclude that ZSTD remains free of patents (?) - suggesting this "The license granted hereunder will terminate, automatically and without notice (...)" from PATENTS file has no legal meaning (?) --- Anyone care to validate how true that statement is?

I imagine zstd's license will hamper its corporate adoption, especially among the big players. The license granted hereunder will terminate, automatically and without notice, if you (or any of your subsidiaries, corporate affiliates or agents) initiate directly or indirectly, or take a direct financial interest in, any Patent Assertion: (i) against Facebook or any of its subsidiaries or corporate affiliates... https://github.com/facebook/zstd/blob/dev/PATENTS

What use case do you have in mind for packaging dictionaries with archives? There is an ongoing discussion about a jump table format that could contain dictionary locations [1]. [1] https://github.com/facebook/zstd/issues/395

It depends on the "window size" used by the compressor. According to https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi..., the window size can be anywhere from 1KB to 1.875TB. The table mapping compression levels to window sizes is at https://github.com/facebook/zstd/blob/15a7a99653c78a57d1ccbf.... The first element is the window size. I'm pretty sure you raise 2 to that power to get the window size. As you can see, level 1 is 2^18, which is 256KB - far larger than the minimum of 1KB. Lowering it will make compression faster at the expense of compression ratio. For comparison, zlib's max window size is 32KB. That's one reason its compression ratio is so limited.