zstd

I don't think so? It's only seekable with an additional index [1], just like most other compression schemes. Having an explicit standard for indices is definitely a plus though. [1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...

The latest Zstandard exposes several parameters which are useful for reducing time to first byte latency in web compression. They make Zstandard cut the compressed data into smaller blocks, e.g. 4 KB, with the goal to fit a compressed block within a small number of packets, so the browswer can start to decompress without waiting for a full 128 KB block to be sent. These parameters are described in the v1.5.6 release notes [0]. ZSTD_c_targetCBlockSize is the most notable, but ZSTD_c_maxBlockSize can also be used for a lower CPU cost but larger compressed size. Are you using these features at Cloudflare? If you need any help using these, or have any questions, please open an issue on Zstandard's GitHub! [0] https://github.com/facebook/zstd/releases/tag/v1.5.6

agree, - in the benchmark "zstd 0.6.0" ( Apr 13, 2016 ) - vs. latest zstd v1.5.6 ( Mar 30, 2024 https://github.com/facebook/zstd/releases )

So to clarify: my suggested point of comparison was replacing QOI + 7Zip of GP with QOIR + zstd. QOIR already compresses better than QOI before the LZ4 pass, and zstd compresses faster than 7zip and often better. On top of that you can put zstd in the header option when streaming data on a browser so you don't need to increase the JS bundle or whatever if the use case is the web. So that's basically a guaranteed net improvement all around. Second of all, the speed/compression trade-off with zstd can be tuned a lot. The "half as fast as LZ4" stat is for the fastest setting, but for the proposed comparison point of 7zip a slower setting with better compression ratio is likely perfectly fine. [0] https://github.com/facebook/zstd/tree/dev?tab=readme-ov-file...

Zstd is a personal project? Surely it's not by accident in the Facebook GitHub organization? And that you need to sign a contract on code.facebook.com before they'll consider merging any contributions? That seems like an odd claim, unless it used to be a personal project and Facebook took it over (https://github.com/facebook/zstd/blob/dev/CONTRIBUTING.md#co...)

> For such small string, any statistical compression won't have a smaller size That is a statement you can make, but you have to actually demonstrate this is true against zstd's --train mechanism [1] which generates a more optimized "small string" dictionary based on representative training data precisely for this use-case. [1] https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md > those are used for different scenarios. zstd is used for data compression, the meta string encoding here is used for meta meta encoding Not sure what this means. The motivating use-case described for the alternate string encoding is precisely compression.

> In such cases, the serialized binary are mostly in 200~1000 bytes. Not big enough for zstd to work You're not referring to the same dictionary that I am. Look at --train in [1]. If you have a training corpus of representative data, you can generate a dictionary that you preshare on both sides which will perform much better for very small binaries (including 200-1k bytes). It's the same kind of technique but zstd's mechanism is absolutely general whereas purpose-built non-entropy based dictionaries are more limited & less likely to outperform. If you want maximum flexibility (i.e. you don't know the universe of representative messages ahead of time or you want maximum compression performance), you can gather this corpus transparently as messages are generated & then generate a dictionary & attach it as sideband metadata to a message. You

You can already pre-train compression on text without using an LLM: $ curl https://www.gutenberg.org/cache/epub/11/pg11.txt > text.txt $ split -n 500 text.txt trainpart. Using a normal compression algorithm: $ zstd --train trainpart.* -o dictionary Save dictionary of size 112640 into file dictionary $ zstd -vD dictionary text.txt *** Zstandard CLI (64-bit) v1.5.5, by Yann Collet *** text.txt : 15.41% ( 170 KiB => 26.2 KiB, text.txt.zst) For this example, ZSTD warns that the dictionary training set is 10X-100X too small to be efficient. Realistically, I guess you'd train it over E.G. the entire Gutenberg library. Then you can distribute specific books to people who already have the dictionary. Or: $ curl -L https://archive.org/download/completeworksofl1920carr/completeworksofl1920carr_h

This VFS will read a sqlite file after it has been compressed using [zstd seekable format](https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...). Built to support read-only databases for full-text search. Benchmarks are provided in README.

My point is, it's already possible to use multiple independently compressed (and decompressable) frames with zstd if you really want to. It's even in the zstd repo, under a "contrib" implementation https://github.com/facebook/zstd/blob/87af5fb2df7c68cc70c090... That does, of course, require that you compress it into multiple frames to begin with, which could be a problem if you don't control the source of the compressed files, because the default is a single frame. In theory if everyone used pzstd to compress their files, it would be strictly superior to BZ2 in nearly every circumstance. As it is, you do have to go out of your way to do that. But I don't think that necessarily means the single-frame choice by default is a bad tradeoff. It's better in most circumstances. And if they do eventually figure out a reasonably efficient way to

That's possible with pzstd in any case. zstd upstream has a plan to eventually support parallel decompression natively but hasn't prioritized it given the complexity and lack of immediate need. https://github.com/facebook/zstd/issues/2499#issuecomment-78...

For a benchmark on a standard set: https://github.com/inikep/lzbench/blob/master/lzbench18_sort... Of course, you may get different results with another dataset. gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s] zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s] NB1: The default for zstd is -3, but the table only had -2. The difference is probably small. The range is 1-22 for zstd and 1-9 for gzip. NB2: The default program for gzip (at least with Debian) is the executable from zlib. With my workflows, libdeflate-gzip iscompatible and noticably faster. NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay a 10× penalty on decompression. xz -9 [ratio=23%

There is an issue tracking this with a bunch of links to discussions about it, but they continue to not have time it seems. https://github.com/facebook/zstd/issues/3100 This was the first place my mind went when I saw this Content-Encoding announcement, so I ran and re-checked the issue :(.

Citation needed? https://github.com/facebook/zstd/commits/dev/?author=jiaT75

Yes, but they also work for megacorp Facebook, and according to https://github.com/facebook/zstd/graphs/contributors, 300+ other contributors have made 4500+ commits to the zstd repo. It's not quite as small-scale as 90's-style shareware was.