nimble

The wasm decoder thing was also done in Anyblox. https://github.com/AnyBlox https://gienieczko.com/anyblox-paper Has nimble/velox had any better luck lately? I forget what stories someone shared, but, it seemed to have such big intent, then real trouble actually getting released. I want to say someone was saying the lawyers ended up not letting a lot of the work get released. Nimble is the one competitor benchmarked against here that beats them, and is also extensible (to some degree?), so I'd love to know how things have gone for the past 6-12 months for nimble/velox. https://news.ycombinator.com/item?id=39995112 https://github.com/facebookincubator/nimble/ https://materializedview.io/p/nimble-and-lance-parquet-kille...

It's important in a book treating an emerging field (data eng for LLMs) to mention emerging categories related to it such as storage formats purpose built for the full ML lifecycle. Lance[1] (the format, not just LanceDB) is a great example, where you have columnar storage optimized for both analytical operations and vector workloads together with built-in versioning for dataset iteration. Plus (very important) random access, which is important for stuff like sampling and efficient filtering during curation but also for working with multimodal data, e.g. videos. Lance is not alone, vortex[2] is another one, nimble[3] from Meta yet another one and I might be missing a few more. [1] https://github.com/lance-format/lance [2] https://vortex.dev [3] https://github.com/facebookincubator/nimble

As others said, Vortex is complementary to the table Formats you mentioned. There are other formats though that it can be compared to. The Lance columnar format is one: https://github.com/lancedb/lancedb And Nimble from Meta is another: https://github.com/facebookincubator/nimble Parquet is so core to data infra and widespread, that removing it from its throne is a really really hard task. The people behind these projects that are willing to try and do this, have my total respect.

The backstory is complicated. The plan was to establish a consortium between CMU, Tsinghua, Meta, CWI, VoltronData, Nvidia, and SpiralDB to unify behind a single file format. But that fell through after CMU's lawyers freaked out over Meta's NDA stuff to get access to a preview of Velox Nimble. IANAL, but Meta's NDA seemed reasonable to me. So the plan fell through after about a year, and then everyone released their own format: → Meta's Nimble: https://github.com/facebookincubator/nimble → CWI's FastLanes: https://github.com/cwida/FastLanes → SpiralDB's Vortex: https://vortex.dev → CMU + Tsinghua F3: https://github.com/future-file-format/f3 On the research side, we (CMU + Tsinghua) weren't interested in developing new encoders and instead wanted to focus on the WASM embedding part. The original idea came as a suggestion from Hannes@DuckDB

Off the top of my head: - Vortex https://github.com/vortex-data/vortex - Lance https://github.com/lancedb/lance - Nimble https://github.com/facebookincubator/nimble There are also a bunch of ideas coming out of academia, but I don't know how many of them have a sustained effort behind them and not just a couple of papers

Lance (from LanceDB folks), Nimble (from Meta folks, formerly known as Alpha); I think there are a few others https://github.com/lancedb/lance https://github.com/facebookincubator/nimble

Nice to see methodology here. Ideally Lancedb lance v2 and nimble would also both be represented here. It feels like there's huge appetite to do better than Parquet; ideally work like this would help inform where we go next. https://blog.lancedb.com/lance-v2/ https://github.com/facebookincubator/nimble