lance

Would there be an advantage to reusing SIMD-optimized libraries for this? Polars Rust is built on Arrow and packed_simd. pola-rs/polars: https://github.com/pola-rs/polars polars - Rust API docs: https://docs.pola.rs/api/rust/dev/polars/#simd lancedb's data format; Lancedb/lance works with [Pandas, DuckDB, Polars, PyArrow,]; https://github.com/lancedb/lance Narwhals' df interface (Python) https://narwhals-dev.github.io/narwhals/ substrait's portable query plans: https://substrait.io/ , https://github.com/ibis-project/ibis-substrait Arrow RecordBatch, https://news.ycombinator.com/item?id=45495738#45546244 cargo-fuzz, TLA+

It's important in a book treating an emerging field (data eng for LLMs) to mention emerging categories related to it such as storage formats purpose built for the full ML lifecycle. Lance[1] (the format, not just LanceDB) is a great example, where you have columnar storage optimized for both analytical operations and vector workloads together with built-in versioning for dataset iteration. Plus (very important) random access, which is important for stuff like sampling and efficient filtering during curation but also for working with multimodal data, e.g. videos. Lance is not alone, vortex[2] is another one, nimble[3] from Meta yet another one and I might be missing a few more. [1] https://github.com/lance-format/lance [2] https://vortex.dev [3] https://github.com/facebookincubator/nimble

It looks like Narwhals; "Narwhals and scikit-Lego came together to achieve dataframe-agnosticism" https://news.ycombinator.com/item?id=40950813 : > Narwhals: https://narwhals-dev.github.io/narwhals/ : >> Extremely lightweight compatibility layer between [pandas, Polars, cuDF, Modin] > Lancedb/lance works with [Pandas, DuckDB, Polars, Pyarrow,]; https://github.com/lancedb/lance SymPy has Solvers for ODEs and PDEs and other libraries do convex optimization. SymPy also has lambdify to compile from a relatively slow symbolic expression tree to faster 'vectorized' functions From https://news.ycombinator.com/item?id=40683777 re: warp : > sympy.utilities.lambdify.lambdify() https://github.com/sympy/sympy/blob/master/sympy/utilities/l... : >>> """Convert a SymPy expression into a function that allows for fast numeric evaluation""" [with e.g. the

Off the top of my head: - Vortex https://github.com/vortex-data/vortex - Lance https://github.com/lancedb/lance - Nimble https://github.com/facebookincubator/nimble There are also a bunch of ideas coming out of academia, but I don't know how many of them have a sustained effort behind them and not just a couple of papers

Lance (from LanceDB folks), Nimble (from Meta folks, formerly known as Alpha); I think there are a few others https://github.com/lancedb/lance https://github.com/facebookincubator/nimble

How does this relate to https://github.com/lancedb/lance

Narwhals: https://narwhals-dev.github.io/narwhals/ : > Extremely lightweight compatibility layer between [pandas, Polars, cuDF, Modin] Lancedb/lance works with [Pandas, DuckDB, Polars, Pyarrow,]; https://github.com/lancedb/lance It may already be supported through pandas?

https://news.ycombinator.com/item?id=40115099 : > lancedb/lance is faster than pandas with dtype_backend="arrow" and has a vector index From https://github.com/lancedb/lance : > Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, > [...] Vector Search > Comparison of different data formats in each stage of ML development cycle: Lance, Parquet & ORC, JSON & XML, TFRecord, Database, Warehouse What can ChromaDB do that lancedb can't, and will either work in WASM? From https://github.com/chroma-core/chroma : > By default, Chroma uses Sentence Transformers to embed for you but you can also use OpenAI embeddings, Cohere (multilingual) embeddings, or your own.

How do Parquet, Lance, and Nimble compare? lancedb/lance: https://github.com/lancedb/lance : > Modern columnar data format for ML. Convert from Parquet in 2-lines of code for 100x faster random access, a vector index, data versioning, and more. Compatible with pandas, DuckDB, Polars, and pyarrow Can the optimizations in lance be ported to other formats without significant redesign?

Parquet has been the lakehouse file format of choice for nearly half a decade. But we are starting to see other contenders that are optimized more for lower latency like lance https://github.com/lancedb/lance