rapidgzip

slightly faster than cat when comparing the piped bandwidth! E.g., compare these additional benchmarks: time cat 4GiB-base64.gz | wc -c # real ~1.06 s -> ~3.1 GB/s time fcat 4GiB-base64.gz | wc -c # real ~0.41 s -> ~8.0 GB/s time rapidgzip -o /dev/null -d 4GiB-base64.gz # real ~0.68 s -> ~6.5 GB/s fcat is an alternative cat implementation that uses vmsplice to speed up piping. According to the ReadMe it currently is broken, but it works fine on my system and piping it to md5sum yields consistent results [2]. So, at this point, I/O and actually also allocations have become a limiting factor and if you want full speed, you would have to interface with the rapidgzip library interface directly (in C++ or via the Python bindings) and process the decompressed data in memory. The project ReadMe contains further benchmarks with Silesia and FASTQ data and scaling up to 128 cores, for which rapidgzip achieves 12 GB/s for Silesia and 24 GB/s when an index has been created with --export-index and is used with --import-index. It can also be tested with ratarmount 0.14.0, which now uses rapidgzip as a backend by default for .gz and .tar.gz files [3]. [0] https://news.ycombinator.com/item?id=32366959 [1] https://dl.acm.org/doi/10.1145/3588195.3592992 [2] https://github.com/mre/fcat [3] https://github.com/mxmlnkn/ratarmount

t base64-encoded random data is a weird test case but it has its pros because it is a kind of minimal benchmark for raw Huffman decoding speed without (many) LZ references that need to be resolved. I redid the benchmark as outlined above with the three test files that I am also using for my extended benchmarks [0]: 4GiB-base64.gz -> libdeflate: 20.5 s, igzip: 9.4 s, rapidgzip: 1.5 s 20xsilesia.tar.gz -> libdeflate: 5.4 s, igzip: 6.6 s, rapidgzip: 1.8 s 10xSRR22403185_2.fastq.gz -> libdeflate: 5.8 s, igzip: 5.5 s, rapidgzip: 1.9 s File Sizes: Compressed -> Uncompressed: 4GiB-base64.gz : 4294967296 -> 3263906203 20xsilesia.tar.gz : 1364776140 -> 4239155200 10xSRR22403185_2.fastq.gz : 970458140 -> 3618153020 In conclusion, it seems that it highly depends on the test case and the one I tested to, too quickly, check my statement is one of the outliers. [0] https://github.com/mxmlnkn/rapidgzip#scaling-benchmarks-on-2...

ompressed blocks, the false positive rate is roughly one per 500 KB, however non-compressed blocks can basically be memcpied or skipped over and then the next deflate header can be checked without much latency. On the other hand, for dynamic blocks, the whole block needs to be decompressed first to find the next one. So the much higher false positive rate for non-compressed blocks doesn't introduce that much overhead. Furthermore, you need to consider that the block finder is basically only applied to the first ~0-128 KiB or so data of each 4 MiB large block (until it finds a valid one). So, in terms of real input archive size, the effective false positive rate is smaller by a factor of 32. I have some profiling built into rapidgzip, which is printed with -v, e.g., rapidgzip -v -d -o /dev/null 20xsilesia.tar.gz : Time spent in block finder : 0.227751 s Time spent decoding : 20.4015 s Time spent allocating and copying : 2.71328 s Time spent applying the last window : 1.39377 s Replaced marker bytes : 1 GiB 497 MiB 311 KiB 503 B The block finder adds about 1% overhead and the marker replacement about 7% overhead. It probably would be nice to add a statistic for the false positive rate to the -v output.