zstd

Zstandard is pretty nice: https://github.com/facebook/zstd

Zstd fully supports binary deltas. If you're computing a delta between `old` and `new` you would do this to round trip it: zstd --patch-from=old new -c | zstd -d --patch-from=old Using the library this can be achieved by using the old file as a dictionary. If you have very large files (MB-GB range) you will also want to enable long range mode, otherwise zstd might not be able to find the repetition. --patch-from on the CLI will manage this automatically. The release notes from zstd-1.4.5 [0] contain some details about the speed/size tradeoff of "patch-from" mode, compared to other delta engines. There is also a wiki page detailing how to use zstd as a delta engine [1]. [0] https://github.com/facebook/zstd/releases/tag/v1.4.5 [1] https://github.com/facebook/zstd/wiki/Zstandard-as-a-patchin...

Yeah, I've experimented. It is actually surprisingly close to native performance! A testament to the wasm engine, I guess. There's discussion on this topic here: https://github.com/facebook/zstd/issues/1355

https://github.com/facebook/zstd/issues/395#issuecomment-535... If anyone from Facebook is reading, please look at making zstd seekable. There is already experimental support but it needs to be standardized. This could open up really interesting new use-cases [see my comment linked above], where currently we have to use much slower xz/lzma instead.

There's a web assembly version in the repository : https://github.com/facebook/zstd/tree/dev/contrib/single_fil... Could be worth checking

Zstandard also has experimental support for seekable (and thus parallel) decompression, but it isn't included in distributed binaries. See: https://github.com/facebook/zstd/issues/395

Everybody's needs are different, but zstd is so fast the lack of multithreaded decompression doesn't matter very often. I've only ever found parallel decompression useful with bz2 files (which are insanely CPU-intensive to decompress), using lbzip2. Zstd also has experimental code in the `contrib` dir for parallel (and, more importantly, seekable) decompression: https://github.com/facebook/zstd/issues/395

In this case it could use the Linux crypto API, which already has ZSTD support and provides compress/decompress functions. But that is exported as GPLv2 so ZFS needs to do its own thing. And idk if the API is sufficient w.r.t. to e.g. workspace management/reuse. W.r.t. to ZSTD: The usual thing is to provide a zlib compatible API, which ZSTD does ( https://github.com/facebook/zstd/tree/dev/zlibWrapper ). But the ZSTD zlib compatibility layer causes lower performance, so it is better to use it directly. Maybe all future compression algorithms can provide a ZSTD compatible API, so we have to do less work in the future?

- Svelte, a compiled front end JavaScript framework https://svelte.dev/ - Nix, a functional package manager https://nixos.org/nix/ basically, to install stuff, instead of running `sudo apt install your_package` you edit `/etc/nixos/configuration.nix` and then regenerate your OS from the config stored in that file using `nixos-rebuild switch`. The benefit is that then you can take that file to a new computer with a new installation of NixOS and regenerate much of the state of your system just with that one file. - QUIC and HTTP/3. I think you have to read the RFCs starting with HTTP/2 to understand it, but really I just mean that you should be aware of them. I think HTTP/3 should start to really exist some time this year and you should consider enabling it in nginx once it becomes available. - Rust - A dependently typed programming languag

Brotli has wide browser support (https://caniuse.com/#feat=brotli) and comes closer to zstd in compression ratio and compression speed, but its decompression speed is significantly lower and closer to zlib. https://github.com/facebook/zstd#benchmarks AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at least Cloudflare seems to serve it by default (EDIT: must be enabled per-site https://support.cloudflare.com/hc/en-us/articles/200168396-W...)

I wish zstd supported seeking and partial decompression (https://github.com/facebook/zstd/issues/395#issuecomment-535...). We could then use it for hosting disk images as it would be a lot faster than xz which we currently use.