pigz

Parallel compression (pigz [0]) and decompression (rapidgzip [1]), for one. When you're dealing with multi-TB files, this is a big deal. [0]: https://github.com/madler/pigz [1]: https://github.com/mxmlnkn/rapidgzip

You can grab the version from the solaris userland repo I linked and use it without me completing a homework assignment. Just grab the pigz-2.3.4 source then apply the patches from [1] in the proper order. Maybe some of them aren't needed for non-Solaris. 1. https://github.com/oracle/solaris-userland/tree/master/compo... I thought I had opened a PR for that a long while ago, but it doesn't show up on github these days. In any case, I did ask Mark Adler to review it. It was never a priority, then the code changed in ways that I don't really want to deal with. While looking through the PRs, I noticed a PR for Blocked GZip Format (BGZF) [2]. That's very interesting, and perhaps suggests that bgzip is a tool you would be interested in. 2. https://github.com/madler/pigz/pull/19

Would pigz's parallel compression prevent that failure case, at least limiting it to its block size of a default 128K? https://github.com/madler/pigz/blob/master/pigz.1#L43-L45 That would be a nice extra benefit, besides the speedup from being multithreaded. (I assume zstd also does multithreading but for those stuck with gzip, this is a drop-in replacement.) Edit: bzip2 apparently does the same, "bzip2 compresses files in blocks, usually 900 kbytes long. Each block is handled independently. If a media or transmission error causes a multi-block .bz2 file to become damaged, it may be possible to recover data from the undamaged blocks in the file." (--man bzip2)

Assuming your decompressor is as capable as gzip(1) is, you don't need to write your own compressor or rely on block alignment, because: • pigz (https://github.com/madler/pigz) has -0 for no-compression compression; • you can concat multiple gzip files together. $ echo foo | gzip > foobar.gz $ echo bar | pigz -0 >> foobar.gz $ echo baz | gzip >> foobar.gz $ hd foobar.gz | grep -C1 -w bar 00000010 a8 65 32 7e 04 00 00 00 1f 8b 08 00 70 bf 8c 63 |.e2~........p..c| 00000020 00 03 01 04 00 fb ff 62 61 72 0a e9 b3 a2 04 04 |.......bar......| 00000030 00 00 00 1f 8b 08 00 00 00 00 00 00 03 4b 4a ac |.............KJ.| $ gzip -d < foobar.gz foo bar baz This also solves the CRC problem, because pigz computes the checksum for you.

The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.

This is a low-effort post, showing basic CLI usage and nothing else. Pigz does not actually decompress in parallel, as the author implies, underscoring the importance of benchmarking things — especially when you're only considering them for performance reasons. See: https://github.com/madler/pigz/issues/36 For compression, pigz is only useful for compressing a single large file in a scenario where you must use gzip. For large collections of files, wrapping normal gzip with GNU Parallel would be faster. In scenarios where you control both compression and decompression, you'd be better off picking a more modern compression algorithm than gzip. ZStandard is almost always the right choice, but this benchmark is a great resource for comparing other options: https://quixdb.github.io/squash-benchmark/

GitHub: https://github.com/Piezoid/pugz. From the readme: "Contrary to the pigz program which does single-threaded decompression (see https://github.com/madler/pigz/blob/master/pigz.c#L232), pugz found a way to do truly parallel decompression. In a nutshell: the compressed file is splitted into consecutive sections, processed one after the other. Sections are in turn splitted into chunks (one chunk per thread) and will be decompressed in parallel. A first pass decompresses chunks and keeps track of back-references (see e.g. our paper for the definition of that term), but is unable to resolve them. Then, a quick sequential pass is done to resolve the contexts of all chunks. A final parallel pass translates all unresolved back-references and outputs the file."

pigz doesn't help much with decompression: https://github.com/madler/pigz/issues/36#issuecomment-249041... Decompression can’t be parallelized, at least not without specially prepared deflate streams for that purpose. As a result, pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances.

The max window size for zlib is 32 KB, so I don't think the default sharding at 128 KB would change much. You can pass the -b parameter if you find out a bigger shard works better on your data. If you are looking for details of the design of pigz, there is a very well-documented overview in the source of pigz.c: https://github.com/madler/pigz/blob/master/pigz.c#L187