zstd

> 8MB I believe for high levels Yep, i found it in the source here: - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... Also, zstd docs say: > Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor. That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

> compression levels -1, 1 or 2 Are the available compression levels not 1-22? Though that might depend on the specific library used - the official lib at least uses 1-22: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...

> I don't see how that's possible if the cache no longer stores the uncompressed data. Zstd has a seekable format for frames, similar to pigz --independent works. [1] - https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...

There's nothing about random access at that link. There's this, but it doesn't seem to be getting much traction: https://github.com/facebook/zstd/tree/dev/contrib/seekable_f...

First, How fast is the Zstd decoder in python at runtime? If rust and python are essentially the same cost, then chose rust. Second, I am surprised that python scored slightly better than rust. My own experience is that, when programming python, Claude would spend so much more time dealing with the code not working at runtime, while for any given rust problem, rust would likely fail at compile time, iterating faster and taking less tokens. Some tasks in python it just completely failed at, writing awful garbage. I suspect that is because there is much more awful garbage written in python. (I was trying to write an AI harness. Python seemed like the obvious choice. It was decidedly not). But in this article, python took slightly less time and tokens than rust for both experiments. I asked Claude: could you write a decoder, from memory, in

> > Pareto optimality is a situation where no […] preference criterion can be better off without making at least one […] preference criterion worse off […]. (Omissions theirs.) Wasn't that zstandard's stated goal? Not very surprising that it has this property, also considering it's much newer (2015) than the established tools like gzip (1992), bzip2 (1996), LZMA as used by xz utils (1999) Edit: https://github.com/facebook/zstd/blob/4856a00164c1d7b947bd38... initial commit indeed states it's meant to have good ratios and good (de)compression speeds as compared to other tools, without sacrificing one for the other (»"Standard" translates into everyday situations which neither look for highest possible ratio (which LZMA and ZPAQ cover) nor extreme speeds (which LZ4 covers).«). So Pareto by design, just not using that name

lz4 can still be the right choice when decompression speed matters. It's almost twice as fast at decompression with similar compression ratios to zstd's fast setting. https://github.com/facebook/zstd?tab=readme-ov-file#benchmar...