arrow-rs

I had no idea what Arrow is: https://arrow.apache.org or arrow-rs: https://github.com/apache/arrow-rs

It's ever discussed in https://github.com/apache/arrow-rs/issues/3888 for comparing object_store in Apache Arrow to the APIs provided by Apache OpenDAL. Briefly, Apache OpenDAL is a library providing FS-like APIs over multiple storage backends, including S3 and many other cloud storage. A few database systems, such as GreptimeDB and Databend, use OpenDAL as a better S3 SDK to access data on cloud storage. Other solutions exist to manage filesystem-like interfaces over S3, including Alluxio and JuiceFS. Unlike Apache OpenDAL, Alluxio and JuiceFS need to be deployed standalone and have a dedicated internal metadata service.

rust_serialization_benchmark: https://github.com/djkoloski/rust_serialization_benchmark apache/arrow-rs: https://github.com/apache/arrow-rs From https://arrow.apache.org/faq/ : > How does Arrow relate to Flatbuffers? > Flatbuffers is a low-level building block for binary data serialization. It is not adapted to the representation of large, structured, homogenous data, and does not sit at the right abstraction layer for data analysis tasks. > Arrow is a data layer aimed directly at the needs of data analysis, providing a comprehensive collection of data types required to analytics, built-in support for “null” values (representing missing data), and an expanding toolbox of I/O and computing facilities. > The Arrow file format does use Flatbuffers under the hood to serialize schemas and other metadata needed to implement the Arrow binary I

One of the interesting components of Polars that I've been watching is the use of the Apache Arrow memory format, which is a standard layout for data in memory that enables processing (querying, iterating, calculating, etc) in a language agnostic way, in particular without having to copy/convert it into the local object format first. This enables cross-language data access by mmaping or transferring a single buffer, with zero [de]serialization overhead. Something genius and obvious in hindsight. For some history, there has been a bit of contention between the official arrow-rs implementation and the arrow2 implementation created by the polars team which includes some extra features that they find important. I think the current status is that everyone agrees that having two crates that implement the same standard is not ideal, and they are

It's built around the arrow-rs library, which we've contributed to significantly: https://github.com/apache/arrow-rs

Oh, and for anyone interested in pitching in on the Rust implementation, there's an issue logged here along with some discussion: https://github.com/apache/arrow-rs/issues/1323

Just a heads up - ballista has been donated to apache arrow, but they also broke the Rust arrow libraries into separate repos [0][1] in a break from the mono-repo model. Read more at the announcement[3] or the merge request[4]. [0] https://github.com/apache/arrow-datafusion [1] https://github.com/apache/arrow-rs [2] https://arrow.apache.org/blog/2021/05/04/rust-dev-workflow/ [3] https://github.com/apache/arrow/pull/10096