FiniteStateEntropy

Interesting example! I've been learning AVX512 by using it to optimize Huffman coding. I found _mm512_permutexvar_epi8 and used it to do byte-indexed lookups, but _mm512_permutex2var_epi8 means I can get by with 2 shuffles and 1 comparison instead of 4 shuffles and 3 comparisons. In the end, on my CPU (AMD 9950x), changing to _mm512_permutex2var_epi8 only sped up compression by ~2%. Compared to Huff0[1] (used by Zstd), my AVX512 code is currently ~40% faster at both compression and decompression. This requires using 32 datastreams instead of 4 used by Huff0. [1] https://github.com/Cyan4973/FiniteStateEntropy

It's obsolete. DEFLATE format is limited to 32KB LZ window with huffman coding. Zstd can use a much larger window (8MB recommended) and a much better entropy coder: https://github.com/Cyan4973/FiniteStateEntropy

Sounds similar to: https://github.com/Cyan4973/FiniteStateEntropy https://arxiv.org/abs/1311.2540 > The modern data compression is mainly based on two approaches to entropy coding: Huffman (HC) and arithmetic/range coding (AC). The former is much faster, but approximates probabilities with powers of 2, usually leading to relatively low compression rates. The latter uses nearly exact probabilities - easily approaching theoretical compression rate limit (Shannon entropy), but at cost of much larger computational cost. Asymmetric numeral systems (ANS) is a new approach to accurate entropy coding, which allows to end this trade-off between speed and rate: the recent implementation [1] provides about 50% faster decoding than HC for 256 size alphabet, with compression rate similar to provided by AC. This advantage is due to being simpler than A

zstd uses ANS [1][2] to encode its (literal length, match length, offset) triples. [1] https://github.com/Cyan4973/FiniteStateEntropy [2] https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi...

Let's see: - Wavpack [1], which is a rough contemporary but offers three tiers of presets (normal scale, high scale, extra high scale) and an innovative (and optional) lossy/hybrid mode - TAK [2] which compressed better and decoded faster than either, but was initially closed-source until the dev was persuaded to open it up - LossyWAV [3] which isn't lossless but chops off least-significant-bits while using noise shaping to pre-process audio and make it compress better when fed to a lossless compressor Most of these developments were first publicized on Hydrogenaudio. But as for innovations in the last two years, not that I'm aware. [1] http://wiki.hydrogenaud.io/index.php?title=WavPack [2] http://wiki.hydrogenaud.io/index.php?title=TAK [3] http://wiki.hydrogenaud.io/index.php?title=LossyWAV EDIT (for some more background): generally in l

I have been waiting for this to hit 1.0 and more importantly get popular so that I can use it everywhere. I am really a fan of Yann Collet's work. These are extremely impressive work specially when you consider that lz4 seems to be better than snappy (by google) and zstandard from LZFSE (from apple). I think he is the first one to write a practical fast arithmetic coder using ANS. And look at how his huffman implementation blazes past zlib huffman though compresses less than FSE [0]. I also like reading his blog posts. While a lot of them goes over my head I can generally make a sense of what he is trying and why something's working despite the complexity. [0] https://github.com/Cyan4973/FiniteStateEntropy

Well, many people nowadays pretend that, as if they read Jarek's paper, understood it and came out with a genuine implementation of their own. But Jarek's ANS paper was first out in 2007, and almost no one paid attention to it, because it was plain inscrutable. Many years later, it took an individual to create FSE (https://github.com/Cyan4973/FiniteStateEntropy), to prove that it could be transformed into something actually useful and competitive. Since then, the paper has been updated a few times, borrowing a few points from FSE in order to become more readable. But it's still very hard to read. In contrast, FSE code can be copy/pasted. And all of a sudden, lot of versions have popped out over Internet. By pure chance, they all look like derivatives of FSE or Fabian Giesen's rANS, but they pay tribute to Jarek's ANS paper, because quite

Arithmetic coding is very expensive - zstd uses new entropy coding with compression ratio like arithmetic coding, but with Huffman-like speed: https://github.com/Cyan4973/zstd https://github.com/Cyan4973/FiniteStateEntropy http://encode.ru/threads/2078-List-of-Asymmetric-Numeral-Sys...

History rewritten once again... Apple's fse implementation is based on an open-source work from an individual which predates lzfse by about ~20 months. https://github.com/Cyan4973/FiniteStateEntropy It's even less optimized than the original one, due to a few errors that slipped through during lzfse conception. http://encode.ru/threads/2221-LZFSE-New-Apple-Data-Compressi...

This was popularized by Yann Collet's work on https://github.com/Cyan4973/FiniteStateEntropy

This looks very interesting. Before everybody starts making the obvious comparisons, note that this is from the same guy that made LZ4, and he is someone who clearly knows what he's doing. I've been following his work for some time. It looks like this is the evolution of Zhuff, an experimental (closed-source) compressor [1]. It is basically LZ4 followed by a fast entropy coder, specifically FSE [2], that is a flavor of arithmetic coding that is particularly suited for lookup-table based implementations. From a quick look at the source code it seems that the entropy stage uses 3 probability tables, one for literal bytes, one for match offsets, and one for match lengths. This is not dissimilar from gzip (which however uses Huffman). EDIT: from a second look it seems that the LZ77 compression stage is basically LZ4: it uses a simple hash tab

Very nice introduction. Recently I've been following the progress of the Finite State Entropy algorithm[1]. From the readme on the github project [2]: >FSE is a new kind of Entropy encoder, based on ANS theory, from Jarek Duda. >It is designed to compete with Huffman encoder and Arithmetic ones. >While huffman is fast but can only represent probabilities in power of 2 (50%, 25%, etc.) arithmetic coding can represent probabilities with much better accuracy, but requires multiplications and divisions. > FSE solve this dilemma by providing precise probabilities, like arithmetic does, but using only additions, masks and shifts. > This makes FSE faster, on par with Huffman speed, and even suitable for low-power CPU environment. [1]: http://fastcompression.blogspot.com/2013/12/finite-state-ent... [2]: https://github.com/Cyan4973/FiniteStateEntr