TurboBench

TurboBench: The Ultimate Data Compression Benchmark 100+ Codecs. The Powerhouse in a Single Executable — The Compression Lie Detector Tired of misleading benchmarks with I/O overhead, cache effects, and CPU throttling? TurboBench delivers high-precision, zero-overhead results you can actually trust. Why TurboBench Stands Out The only benchmark that includes LzTurbo alongside 100+ popular, latest, and fastest compressors (LZ77, Rolz, BWT, Context Mixing, Entropy Coders, and more). - 100% in-memory testing — no disk I/O to skew your results. Supports multiple files, recursive directories, and smart concatenation of small files into multiblock tests. Built to avoid common pitfalls like cache scenarios and throttling found in other tools. - Unlock peak performance — TurboBench reduces throttling by intelligently controlling the CPU frequency

Yes, the most similar one is indeed Selkie. misa77 and oodle are both supported in turbobench (see: https://github.com/powturbo/turbobench), so it's easy to compare them. Results on the silesia corpus on the same Intel setup described in the README, using turbobench: codec decode ratio encode misa77 -0 5309 MB/s 42.64% 58.2 MB/s misa77 -1 4332 MB/s 39.65% 54.5 MB/s selkie -1 3095 MB/s 43.48% 197 MB/s selkie -2 3013 MB/s 41.80% 163 MB/s selkie -3 3081 MB/s 39.58% 95.8 MB/s selkie -4 3498 MB/s 36.95% 43.3 MB/s selkie -5 2607 MB/s 33.58% 2.97 MB/s lz4 2530 MB/s 47.60% 373 MB/s Note: selkie -4 corresponds to the "Normal" level here.

I'm still unconvinced about this addition. And I don't even dislike Zstandard. The main motivation seems to be that while Zstandard is worse than Brotli at the highest level, it's substantially faster than Brotli when data has to be compressed on the fly with a limited computation budget. That might be true, but I'm yet to see any concrete or even anecdotal evidence even in the issue tracker [1] while there exist some benchmarks where both Zstandard and Brotli are fast enough for the web usage even at lower levels [2]. According to their FAQ [3] Meta and Akamai have successfully used Zstandard in their internal network, but my gut feeling is that they never actually tried to optimize Brotli instead (and I'd like to be proven wrong about this, because that would be a great information then!). In fact, Meta employs the main author of Zstand

Sorry, a digit was missing, but now https://github.com/powturbo/TurboBench/issues/43 Well, a correct benchmarking is not done with special data, but with datasets that represent a large set of distributions. Such datasets are for ex. einwik8/9 for text, silesia for a mixed dataset. As a corner case example, RLE-compressible data is not representative for benchmarking compression libraries. If you provide a link for a dataset 10-100MB, I can verify your claims, because I'm not aware of a dataset where igzip is 2 times faster than libdeflate. In TurboBench there is no I/O or other overhead involved, additionally it's single threaded. It's also possible that you're comparing two different CLI programs, one (igzip) I/O optimized and the other as a simple CLI.

I assume you wanted to link to TurboBench and not that particular issue, which for some reason also contains a link for some car listing? Secondly, did you not see my answer to ebiggers under this comment you replied to? Yes, for Silesia, libdeflate is faster, I can confirm, but there are at least two cases for which igzip is faster and one for which igzip is twice as fast. But yes, it heavily depends on the input data. Edit: I was then wondering why I could not find any igzip benchmarks on the repository's ReadMe and then found https://github.com/powturbo/TurboBench/issues/43 , so I guess this is the one you wanted to link to and the 3 got cut off.

libdeflate compress better and has faster decompression than igzip. See the silesia single core in-memory benchmark here [1] comparing zlib,libdeflate,igzip,... https://github.com/powturbo/TurboBench/issues/4

You can use TurboBench [1] to benchmark libdeflate against zlib, igzip, zlib-ng and others. Download TurboBench from Releases [2] Here Some Benchmarks: - https://github.com/zlib-ng/zlib-ng/issues/1486 - https://github.com/powturbo/TurboBench/issues/43 [1] https://github.com/powturbo/TurboBench [2] https://github.com/powturbo/TurboBench/releases

You can use TurboBench [1] to benchmark igzip, zlib-ng and others. Download TurbBench from Releases [2] Here Some Benchmarks: - https://github.com/zlib-ng/zlib-ng/issues/1486 - https://github.com/powturbo/TurboBench/issues/43 [1] https://github.com/powturbo/TurboBench [2] https://github.com/powturbo/TurboBench/releases

No, No and No. It's algorithmically impossible that zstd is decompressing faster than lz4 by using the same environment. You are comparing the zstd with asynchrounous I/O + multithreading against a single thread simple lz4 cli. Additionally you're using some exotic and not representative hardware. In theory, the only single thread decompression case where zstd can beats lz4 is when you have slow (reading) i/o. lz4 & zstd are mostly used as libraries and real benchmarks are not considering I/O or multithreading. This is what I'm getting with TurboBench & enwik9. lz4 decompressing 2,8x faster than zstd. C Size ratio% C MB/s D MB/s Name 356828015 35.7 400.71 1557.59 zstd 1 416874817 41.7 312.31 1967.66 lzav 509199084 50.9 630.78 4395.08 lz4 1 Dow

LZAV now integrated into TurboBench [1] Download TurboBench for windows from releases [2] Lenovo IdeaPad 5 Pro - Ryzen 6600HS - DDR5 6400 - gcc-13.1 File: silesia.tar 220MB (Mixed text/binary) C Size ratio% C MB/s D MB/s Name 77885207 36.7 43.20 4479.24 lz4 9 81343184 38.4 123.17 4299.68 lz4 3 85664894 40.4 324.11 5171.12 lzturbo 11 88672448 41.8 435.96 2769.77 lzav v1.3 99553305 47.0 914.25 5425.54 lzturbo 10 100883728 47.6 813.76 4667.85 lz4 1 211948544 100.0 16032.92 16023.24 memcpy lzturbo (closed source) included as reference [1] https://github.com/powturbo/TurboBench [2] https://github.com/powturbo/TurboBench/releases

see also Discussion with a brotli developer [2] and "Chrome: Implement zstd content-encoding" [3] : [1] - Benchmark App: https://github.com/powturbo/TurboBench [2] - https://twitter.com/powturbo/status/1678333893626195969 [3] - https://bugs.chromium.org/p/chromium/issues/detail?id=124697...

You'll get significantly better compression by using general purpose compressors. Compressesing your sample file, we get 823 bytes with brotli Download TurboBench and make your own tests: [1] - https://github.com/powturbo/TurboBench