zstd

The Visual Display of Quantitative Information is one of my favorite books! 0 in this case is not really a relevant value (since that would mean transforming the input into something infinitely large). The functional identity value / origin here is 1x. Here's what that looks like [1]. To me, this is a significantly less useful image. But maybe that stems from a lot of comfort with both the subject matter and the detail log-log plots that I work with to evaluate zstd performance, e.g. [2]. [1] https://imgur.com/gU2Gdf6 [2] https://github.com/facebook/zstd/pull/1317#issuecomment-4260...

Ha, that just came up: https://github.com/facebook/zstd/issues/1087

In short, we work with individual teams and projects to evaluate their priorities, and select or build the compression scheme that best addresses their needs. Some of those cases are described in the post, but we wanted to summarize the general kinds of results we see. I did spend a fair amount of time working on a graphic for the post to try to capture the distribution of improvements we've seen across different use cases at Facebook, but it ended up being very hard to interpret / glean anything meaningful from. It's difficult to present rigorous conclusions: the reality of data compression is that every use case is different, with different priorities and data characteristics that cause different compressors to behave differently. Some data (random noise) is totally incompressible, and zstd will do just as poorly as any other algorithm.

Porting is generally very easy. * If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm. * Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm. * The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support. * Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1] [0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper

There is a zlib wrapper included in the project : https://github.com/facebook/zstd/tree/master/zlibWrapper

This may help? https://github.com/facebook/zstd/issues/1155

If you're using lrzip, you should also check out zstd long range mode [0]. It uses a long window (128 MB by default, up to 2 GB), together with an efficient search strategy, and multithreading. For example, a 2 GB window, with 4 threads, at level 10: zstd --long=31 -T4 -10 It should be faster than lrzip + zstd and provide about the same results. [0] https://github.com/facebook/zstd/releases/tag/v1.3.2

I share the sentiment. While the tech is getting faster and faster, our typical ways of handling data are getting worse and worse, eliminating all the gains. Devs are just tolerating slower languages, tools, architectures, methodologies etc. until a wall of some kind is being hit. "With Rust in Lambda, each 1GB file takes about 23 seconds to download and parse. That’s about a 78x speedup compared to each Python Glue worker." There should be no separate "download" and "processing". The whole thing should take exactly as much time as it takes to download the file via stream processing. Also: https://github.com/facebook/zstd ; better ratios, better speed.

The commit that dual-licensed under BSD and GPLv2 also "removed PATENTS clause": https://github.com/facebook/zstd/commit/4f73b3b55d83b082aad1...

So can someone help me untangle how this relates to possible Zstandard patents? Back some time ago, when all public facebook git repositories had a patent "grant" document, there was some angst about what exactly this meant: https://github.com/facebook/zstd/issues/335 -- this was considered "resolved" when facebook decided to distribute the software under GPLv2, though after reading the relevant sections of GPLv2 I can't say I follow why this was supposed to resolve any uncertainty around potential FB patents of zstd. Under BCP79 it would be Facebook's duty to disclose any IPR it has in zstd; there are presently no such disclosures registered: https://datatracker.ietf.org/ipr/search/?draft=&rfc=8478&sub... As far as I can tell, this is good news for people who would like to adopt zstd!

SLZ is cool, but the specific question jyzg raised and my comment is talking about is whether or not any standards should spell out something like "don't use enormous window sizes with the [still hypothetical] zstd Content-Encoding, to avoid causing problems for small-memory clients decompressing your responses." This comes up because zstd (unlike gzip) lets you set your window to be GBs if you want (look for "long-range match finder" in https://github.com/facebook/zstd/releases if you want more context); those window sizes would, of course, not be ideal on a low-end smartphone, for example.

I'd love this added as an HTTP Content-Encoding, like brotli recently was (and gzip has been for a long time). You could get a bit more compression at gzip-like speeds, or have very cheap compression using its fastest modes (hundreds of MB/s/core; see --fast modes at https://github.com/facebook/zstd/releases/tag/v1.3.4). brotli especially helped at the high end (static stuff you'll serve many times and can spend a long time compressing); zstd can be especially helpful nearer the low end (dynamic content you'll serve a single time). On Twitter a designer of Brotli raised the question of whether there should be a standardized limit on RAM required to decode in the HTTP Content-Encoding context, since zstd supports huge window sizes (GBs). I don't feel strongly--clients can refuse to handle responses that take too much RAM to decode, server

How about zstd? https://github.com/facebook/zstd