TurboPFor-Integer-Compression

Gorilla [1] and Gorilla based algos [2] are simply overrated. Store these values as 32 bits floats instead of 64 bits and you get instant 50% reduction without any compression. This is valid for allmost all time series data. Most of time series databases (ex. DuckDB) are storing floating point data as 64 bits. They are reporting some extraordinary compression ratio by using a gorilla/chimp like algorithm. However as shown in this benchmark, lot of time series data (ex. temparature, climate data, stocks,...) don't have more than 1 or 2 fixed decimal digits and can be stored losslessly in 16/32 bits integers. Integer compression [1] algorithms can then be used, which results in significant compression ratio and several times faster than the gorilla like algorithms. TurboGorilla, the fastest Gorilla (or chimp) based algo in c, cannot exceed

Most time-series / analytical databases,... are already using or switching to integer-compression [1] where you can compress/decompress several times (>100GB/s see TurboBitByte in [2]) faster than general purpose compressors. [1] - https://github.com/powturbo/TurboPFor-Integer-Compression [2] - https://github.com/powturbo/TurboPFor-Integer-Compression/is...

I've some remarks regarding the tests and the test data in this survey. - The data used in the benchmark seems very compressible. However floating point data (also time series) is in general not well losslessly compressible and the Gorilla algorithm is not helping so much. - TurboPFor Integer Compression [1] has also an improved and faster Gorilla algorithm TurboGorilla (not shown in the benchmark) - TurboPFor has now a new lz based algorithm "TurboFloat LzX" witch drastically compress time series floating point data (up to 3 times). - TurboPFor supports lossy compression [2] (TurboRazor) with a point wise relative error bound. The preprocessed data can be further compressed with any floating point compression algorithm. - TurboPFor supports floating point conversion to integers. The result data can be compressed with any integer compress

Excellent blog post serie about floating point compression. All those (lossless+lossy) floating point compression algos including 1D/2D/3D SIMD transpose with integrated xor and zigzag delta are available in the TurboPFor [1] Integer Compession package. There is a benchmark with the same dataset available in [2] with a comparison to zfp and blosc with the new bytedelta. TurboPFor functions compress/decompress faster with better compression. The bytedelta described in the blog is suboptimal and might not work with other datasets. Download icapp from [3] and make your own tests with your data. [1] https://github.com/powturbo/TurboPFor-Integer-Compression [2] https://github.com/powturbo/TurboPFor-Integer-Compression/is... [3] https://github.com/powturbo/TurboPFor-Integer-Compression/re...

It might be possible to use integer compression with block offsets like in the TurboPFor [1] demo. TurboPFor offers also direct access to single values in a block. 1 - https://github.com/powturbo/TurboPFor

Elias-fano are overestimated without any proof or comparison benchmarks. Like other integer compression methods, you still need offsets to small blocks for efficient random access. Getting a single value in a block is done with slow branchy bit ops like ctz and bit shifting. That's decoding The fastest elias-fano implementation is in https://github.com/powturbo/TurboPFor and is partially using SIMD

- Timestamps are very easy to compress down to 0.01% - Floats are hard to compress without quantization or lossy compression. The Gorilla algorithm is not helping so much. You can do your own benchmarks with your data with TurboPFor : https://github.com/powturbo/TurboPFor or download icapp including more powerfull algorithms.

1 - [TurboPFor](https://github.com/powturbo/TurboPFor) 2 - [TurboPFor: an analysis](https://michael.stapelberg.ch/posts/2019-02-05-turbopfor-ana...) 3 - [Debian Code Search:Website](https://github.com/Debian/dcs/) 4 - [Debian Code Search: source code](https://github.com/Debian/dcs/)

libfor benchmarked: https://github.com/powturbo/TurboPFor

- Direct Access w/o decompression - Fastest Variable Byte implementation - Novel Variable Simple faster than simple16, better than simple8-b - Scalar Bit Packing decoding as fast as SIMD-Packing - Bit Packing incl. Direct Access/Update w/ zero decompression - Fastest and most efficient SIMD Bit Packing - Fastest SIMD-Elias Fano implementation - Novel TurboPFor (PFor/PForDelta) with direct access or bulk decoding. More efficient than ANY other "integer compression" scheme. - Java Critical Native Interface. Access TurboPFor incl. SIMD from Java as fast as calling from C. ----------------------------------------------------------- * Inverted Index + Intersections - Novel Intersections w/ skip intervals, decompress the min. #blocks - 2000! queries /sec on GOV2 (25 MB docid) on a SINGLE core - Parallel Query Processing on Multicores. 7000! que