compress

This seems as good a thread as any to mention that the gzhttp package in klauspost/compress for Go now supports zstd on both server handlers and client transports. Strangely this was added in a patch version instead of a minor version despite both expanding the API surface and changing default behavior. https://github.com/klauspost/compress/releases/tag/v1.18.4

For Go, we've been using this library which supports ZSTD. https://github.com/klauspost/compress

I didn't know Go reserved some registers for its own use. For libraries that don't use cgo but do use assembly [1], does this mean they have to generate that assembly with a compiler flag that avoids reserved registers? [1] A common optimization technique, like in the superb https://github.com/klauspost/compress

Try using cgo-based zstd package https://github.com/valyala/gozstd . It is usually faster than pure-Go package https://github.com/klauspost/compress/tree/master/zstd

Only if it's backed by security-reviewed and fuzzed asm. Go's gzip implementation is slow enough that klauspost is worth importing, following that pattern, I would probably still use klauspost's zstd even if Go had a slow reference implementation in the standard library. What I'm really hoping for is something as seamless as gzhttp but for zstd. https://github.com/klauspost/compress/tree/master/gzhttp

There's a pure-go zstd at https://github.com/klauspost/compress - it's likely faster than running the upstream zstd under Wazero.

There is a reasonably feature complete implementation of Zstd for Go: https://github.com/klauspost/compress/tree/master/zstd It may not offer the same API 1:1, but it has no interoperability issues that I've encountered. I think no one has bothered to implement it in Rust because most use cases don't mind the added bloat you're talking about, not because implementing it in Rust would be insurmountable. Plus, other comments I've seen suggest that you can actually tune the size of the C zstd library, although I'm not sure if the Rust bindings expose that.

I noticed Github returns "This page is taking too long to load" with status code 502 for https://github.com/klauspost/compress but rest of their urls works fine. Anyone know why would that be the case ? Cloning the repo works perfectly well. git clone https://github.com/klauspost/compress Cloning into 'compress'... remote: Enumerating objects: 6989, done. remote: Counting objects: 100% (6989/6989), done. remote: Compressing objects: 100% (2444/2444), done. remote: Total 6989 (delta 4019), reused 6877 (delta 3956), pack-reused 0 Receiving objects: 100% (6989/6989), 49.89 MiB | 13.50 MiB/s, done. Resolving deltas: 100% (4019/4019), done

Recent versions of zstd definitely don't obsolete LZ4, or else I don't think the author would still be contributing to both... zstd also seems to be optimizing much more, recently, for the large stream of data use cases, versus the fixed-size or smaller records you might get in, say, databases, or filesystems, so I think it's actually becoming less generally useful over time of late. And if you're going to play with Snappy, you might find S2, which was linked on HN relatively recently, interesting. [1] [1] - https://github.com/klauspost/compress/tree/master/s2

Compression method appears to be zstandard and uses https://github.com/klauspost/compress, for those wondering like I was.

VictoriaMetrics [1] is written entirely in Go. By default it uses canonical zstd library for compression (the library is written in C), but it supports pure Go mode when built with `make victoria-metrics-pure`. In this mode it uses zstd implementation written in Go [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics [2] https://github.com/klauspost/compress/tree/master/zstd#zstd

Out of curiosity, why? Zstd performs as well as it does because of the enormous effort that has been invested into it. That necessarily brings complexity. We do actually have a simple “educational” decoder implementation [0], but I wouldn’t recommend its use in production. And then there are two re-implementations in Java [1] and Go [2]. [0] https://github.com/facebook/zstd/tree/dev/doc/educational_de... [1] https://github.com/airlift/aircompressor/tree/master/src/mai... [2] https://github.com/klauspost/compress/tree/master/zstd#zstd