zstd

And the benchmark has been updated : https://github.com/facebook/zstd#benchmarks - lz4 1.9.3 ; zstd 1.5.1 -1 ; zlib 1.2.11 -1 ; ...

I am amazed at the variety of repos shared - by you, as well as others. Thanks! Repo links: https://github.com/facebook/zstd https://github.com/google/brotli https://github.com/ClickHouse/ClickHouse https://github.com/evanw/esbuild https://github.com/figma https://github.com/tailwindlabs/tailwindcss https://github.com/caddyserver/caddy https://github.com/haproxy/haproxy https://github.com/uNetworking/uWebSockets

You can compile only the decompressor of the official ZSTD library to wasm, it is ony 28 kB: https://github.com/facebook/zstd/files/6602821/zstd.zip

Two points worth noting: 1. Gzip is not a suitable compressor for this use case, because it's limited to a 32KB window. So the input can only be correlated with the last 32KB of the reference texts. 2. You can save a great deal in computation by avoiding recompressing the reference texts over and over and over. Some compression algorithms support checkpointing the compression state so that it can be resumed from that point repeatedly ("dictionary-based compression", which is a distinct capability from just streaming compression, which generally can only be continued once). I would personally shill for using Zstandard [0] instead for this purpose. Although I should disclose my bias: I'm a developer of Zstd. A few salient facts: 1. Zstd supports very large windows (up to 128MB, or up to 2GB in long mode). 2. Zstd is much faster than zlib. 3

To me this puts v2 in the same category as MIT/X11 and BSD: archaic licenses that no one should ever use except for the most trivial of projects. Isn't BSD widely accepted to have an implicit patent license? e.g.: https://github.com/facebook/zstd/issues/335

I recently reworked how zstd builds its Huffman decoding tables (not decoding itself) to avoid unpredictable branches and speed the table building up by about ~2x [0]. This is insignificant for large decompressions, but if you're decompressing only a few KB, the table building time can dominate the actual decompression. It sort of goes to show that while Huffman codes have been around for ages, implementations can still improve, especially as the hardware we use changes. [0] https://github.com/facebook/zstd/pull/2271

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is quite profitable. Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.

another contender is zstd: https://github.com/facebook/zstd. It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

Yes. I work at Facebook on data compression. The libraries we maintain (1) are responsible for a non-trivial part of Facebook's overall compute footprint, (2) should basically never fail of their own accord, and (3) have pretty good error monitoring. So my team is operating what is effectively (among other things) a very sensitive detector for hardware failure. And indeed we see examples all the time of blobs that fail to decompress, and usually when we dig in we find that the blob is only a single bit-flip away from a blob that decompresses successfully into a syntactically correct message. I can't share numbers, but, off the top of my head, I think it's the largest source of failures we see. It happens frequently enough that I wrote a tool to automate checking [0]. So yes. It happens. Pretty frequently, in the sense that if you're doing