arrow2

<rant>The official Arrow C++ implementation is just ergonomic warts, full of `const std::shared_ptr<T>&` bs. Trying to use it to manipulate data always give me headache telling apart WTH is an Array, ArrayData, Buffer, and the typed Array interfaces are barely usable. The original official Rust port inherited all the mis-designs too. On the Rust side someone created arrow2 [0] to fix it.</rant> And I'm glad there's a good C++ impl too. [0] https://github.com/jorgecarleitao/arrow2

One of the interesting components of Polars that I've been watching is the use of the Apache Arrow memory format, which is a standard layout for data in memory that enables processing (querying, iterating, calculating, etc) in a language agnostic way, in particular without having to copy/convert it into the local object format first. This enables cross-language data access by mmaping or transferring a single buffer, with zero [de]serialization overhead. Something genius and obvious in hindsight. For some history, there has been a bit of contention between the official arrow-rs implementation and the arrow2 implementation created by the polars team which includes some extra features that they find important. I think the current status is that everyone agrees that having two crates that implement the same standard is not ideal, and they are

The code is written in high performance multi-threaded Rust and uses the [1] Arrow compute framework. We also batch events and target about 32MB of event data per lambda invocations. As a result it can process tens of thousands of events per second per thread, depending on the number of transformations. That said, we are working on performance estimates and a benchmark on some real world data for Matano to help users like you better understand the cost factors. Stay tuned. [1] https://github.com/jorgecarleitao/arrow2