isa-l

Not actually true if gzip refers to libdeflate[1] or isa-l[2] instead of zlib. Does seem sus here, though,—I couldn’t quickly find which decompression code is used in zboot, but I’d expect it to be closer to zlib. Perhaps their implementation of Zstandard is simpler than the reference one? [1] https://github.com/ebiggers/libdeflate [2] https://github.com/intel/isa-l

If you have large gzip files to decompress, I can recommend igzip[1] for fastest single-core decompression or my pet project rapidgzip[2] for effective multi-core decompression, both have simple CLI tools. For using accelerators (QAT, I assume, because the article is about this) from the CLI, then QATzip[3] comes with a command line tool qatzip which can be used as described in the project ReadMe. I didn't test it, though, as I have no QAT-enabled device. [1] https://github.com/intel/isa-l/tree/master/igzip [2] https://github.com/mxmlnkn/rapidgzip [3] https://github.com/intel/QATzip#test-qatzip

Your link seems to compare GNU gzip with zstd. When comparing file formats, I would compare the best software for that file format. igzip: https://github.com/intel/isa-l can decompress consistently faster than GNU gzip. Depending on the file, it decompresses 2-3x faster making it almost as fast as zstd decompression. I have less experience with compression benchmarks. A quick benchmark on Silesia shows igzip to be ~7x faster but it sacrifices 10% of compression ratio for that even on its highest compression setting. It seems to be optimized for speed.

How does it compare perfomance-wise to Intel ISA-L? https://github.com/intel/isa-l

igzip (https://github.com/intel/isa-l) is much faster than gzip or pigz when it comes to decompression, 2-3x in my experience. There is also a Python module (isal) that provides a GzipFile-like wrapper class, for an easy speed-up of Python scripts that read gzipped files. However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. You also need to make sure to use the latest version if you are going to use it in the context of bioinformatics, since older versions choke on concatenated gzip files common in that field.

> (see mary wootters' talks on youtube) https://www.youtube.com/watch?v=Gh578e98qAk > Suppose I want to write a library to find compression polynomials for reed-solomon encoding system (see mary wootters' talks on youtube) for my storage product. I need a LU decomposition algorithm that operates on GF256 Surely you use isa-l[1]. [1] https://github.com/intel/isa-l > Now I want to accelerate the search by running the search on GPUs. GPUs are float oriented so I don't think you'll get the performance you hope for out of 8 bit integer operations. If you have interesting results to share I'd like to read them.

There is also Intel's ISA-L: https://github.com/01org/isa-l It contains optimized Galois Field multiplication, resulting in Reed Solomon (both Vandermonde as well as Cauchy) at multiple GB/s on a modern x86 CPU. Still, I doubt that the Reed Solomon coding speed is the limiting factor in their rebuild time. There is a mention of a 6-day duration, so even with a very slow Reed Solomon implementation ( ~ 100 MB/s) that should not be a bottleneck for a 10 TB drive rebuild (assuming a distributed rebuild approach, not a traditional RAID style rebuild).

Anyone compared this to the intel ISA-L library that uses the various intel specific instructions to accelerate reed solomon encoding? I believe this is what Facebook, Microsoft, and similar organizations are going to avoid Hadoops 3x replication overhead, while still keeping the same degree of protection. https://github.com/01org/isa-l

Intel's ISA-L ended up implementing this method. Their implementation is interesting because they took this further and took advantage of knowledge of the instruction latency to pipeline multiple iterations of this method to achieve some really amazing throughput. For reference, see https://01.org/intel%C2%AE-storage-acceleration-library-open... and the source code at https://github.com/01org/isa-l (see the erasure code folder for details). In general, I've found Prof. Plank's other papers and presentations very interesting, innovative, and accessible.