lrzip

Love this thing! It enabled me to add a “Convert to GIF” for videos in my Clop app (https://lowtechguys.com/clop) in just a few lines of code. The GIF gets encoded at its optimal size by default so I didn’t have to do a second optimization step. I also like that the author spent time to add an easy way to build a static binary, which I need in order to ship gifski inside the app. Compressing all these static binaries so I can ship them in Clop with a <100MB download was not so easy though. I’ve been using the included xz in macOS but I’m switching to lrzip [1] for better compression size and especially speed [1] https://github.com/ckolivas/lrzip

lrzip Long Range ZIP or LZMA RZIP https://github.com/ckolivas/lrzip "A compression utility that excels at compressing large files (usually > 10-50 MB). Larger files and/or more free RAM means that the utility will be able to more effectively compress your files (ie: faster / smaller size), especially if the filesize(s) exceed 100 MB. You can either choose to optimise for speed (fast compression / decompression) or size, but not both."

I typically don't make use of standard archive formats these days for my own file storage. If I want just pure maximum compression, I'll often use a long-range matcher like FreeArc's srep [1] or lrzip [2] combined with either fast-lzma2 [3] using a p7zip fork [4] for multithreading and fast compression or use mcm [5] or zpaq for max ratio with longer compression time. However, my truly preferred way is using dwarfs [6], which features some really good deduplication and (by default) zstd compression while being mountable. Most of my files are highly compressed and easily accessible without needing to full decompress them. I even made a small script to convert and create AppImages that instead use this [7]. Admittedly, I don't make use of PAR2 or anything of the sort, but I could just do that the traditional way if I so wished. [1]: https:/

Best I know of for that is something like lrzip still, but even then it's probably not state of the art. https://github.com/ckolivas/lrzip It'll also take a hell of a long time to do the compression and decompression. It'd probably be better to do some kind of chunking and deduplication instead of compression itself simply because I don't think you're ever going to have enough ram to store any kind of dictionary that would effectively handle so much data. You'd also not want to have to re-read and reconstruct that dictionary to get at some random image too.

There's also lrzip for large files: https://github.com/ckolivas/lrzip

I always find [lrzip](https://github.com/ckolivas/lrzip) is under appreciated when it comes to compression discussions; it doesn't suit all circumstances, but it works really well in the ones it does (we're using it with the nocompress flag and then using zstd, hence why it comes to mind :-) ) Edit: it's not well suited to real-time...

The next default compressor might be lrzip [1] by Con Kolivas; I've only see it a couple of times in the wild so far but for certain files it can increase the compression ratio quite a bit. [1] https://github.com/ckolivas/lrzip # 151M linux-4.14-rc6.tar.gz # GZIP decompression ~$ time gzip -dk linux-4.14-rc6.tar.gz real 0m4.518s user 0m3.328s sys 0m13.422s # 787M linux-4.14-rc6.tar # LRZIP compression ~$ time lrzip -v linux-4.14-rc6.tar [...] linux-4.14-rc6.tar - Compression Ratio: 7.718. Average Compression Speed: 13.789MB/s. Total time: 00:00:56.37 real 0m56.533s user 5m35.484s sys 0m9.422s # 137M linux-4.14-rc6.tar.lrz # LRZIP decompression ~$ time lrzip -dv linux-4.14-rc6.tar.lrz [...] 100% 786.16 / 786.16 MB Average DeCompression Speed: 131.000MB/s Ou