btrblocks

Lance v2 looks interesting. I like their meta-data + container story. Lacking SOTA encoding schemes though. There is also Vortex (https://github.com/fulcrum-so/vortex). That has modern encoding schemes that we want to use. BtrBlocks (https://github.com/maxi-k/btrblocks) from the Germans is another Parquet alternative. Nimble (formerly Alpha) is a complicated story. We worked with the Velox team for over a year to open-source and extend it. But plans got stymied by legal. This was in collaboration with Meta + CWI + Nvidia + Voltron. We decided to go a separate path because Nimble code has no spec/docs. Too tightly coupled with Velox/Folly. Given that, we are working on a new file format. We hope to share our ideas/code later this year.

> CONCLUSION > We introduced BtrBlocks, an open columnar compression format for data lakes. By analyzing a collection of real-world datasets, we selected a pool of fast encoding schemes for this use case. Additionally, we introduced Pseudodecimal Encoding, a novel compression scheme for floating-point numbers. Using our sample-based compression scheme selection algorithm and our generic framework for cascading compression, we showed that, compared to existing data lake formats, BtrBlocks achieves a high compression factor, competitive compression speed and superior decompression performance. BtrBlocks is open source and available at https://github.com/maxi-k/btrblocks.

Corresponding source code: https://github.com/maxi-k/btrblocks