datafusion-ballista

Hi HN, Chew here, CTO at [Alphaus](https://alphaus.cloud/en/). We've been running a similar version of Luna at Alphaus, to cache really huge CSVs and PARQUETs in memory (hundreds of GBs per account across multiple files). It's written in Go, and uses DuckDB and Apache Arrow behind the scenes. We're now at a point where a single VM's memory is really not enough to hold these huge files (the really hi-mem VMs are not yet available in our region) so we're thinking of distributing the cache across multiple hi-mem VMs (with disk spill-over). I've been "itching" to rewrite this software using a non-GC language, and so I thought it's a good timing to do an OSS version of it. I did consider C++ (and Zig) but I think Rust is a good fit here. I've looked at [Ballista](https://github.com/apache/datafusion-ballista) actually as I'm quite familiar wit

I would expect a better contender for Spark to be something that's actually open source, such as https://github.com/apache/datafusion-ballista

But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.

Not super on topic because this is all immature and not integrated with one another yet, but there is a scaled-out rust data-frames-on-arrow implementation called ballista that could maybe? form the backend of a polars scale out approach: https://github.com/apache/arrow-ballista

> Is Rust's compile-time GC about something other than performance somehow? AFAIK, memory safety and language features as RAII is also available in C++, for instance. About the reasons for slow compilation, take a look at https://www.reddit.com/r/rust/comments/xna9mb/why_are_rust_p... Not having a GC is also about not having a runtime as you mention (e.g. nice for creating Python extensions and embedded systems programming) and also more runtime deterministic performance: on that, if I'm not mistaken that was the reason for Discourse switching to Rust and also, e.g.: "the choice of Rust as the main execution language avoids the overhead of GC pauses and results in deterministic processing times" https://github.com/apache/arrow-ballista/blob/main/README.md

Can you explain how this might differ from something like https://github.com/apache/arrow-ballista I've seen several variants of "next-gen" spark, but nowhere have I really seen the different tradeoffs/advantages/disadvantages between them.

Rust has amazing integration with Python through PyO3 [1] so see it like a safe alternative for high performance calculations and operations. The rust data science ecosystem itself is starting to come together with projects like Polars [2] (Pandas alternative), nalgebra [3], Datafusion [4] and Ballista [5]. Of course nowhere near Python but that is hard to beat today. [1] https://github.com/PyO3/pyo3 [2] https://github.com/pola-rs/polars/ [3] https://github.com/dimforge/nalgebra [4] https://github.com/apache/arrow-datafusion [5] https://github.com/apache/arrow-ballista

Just wanted to also give a shout out to Apache DataFusion[0] that IOx relies on a lot (and contributes to as well!). It's a framework for writing query engines in Rust that takes care of a lot of heavy lifting around parsing SQL, type casting, constructing and transforming query plans and optimizing them. It's pluggable, making it easy to write custom data sources, optimizer rules, query nodes etc. It's has very good single-node performance (there's even a way to compile it with SIMD support) and Ballista [1] extends that to build it into a distributed query engine. Plenty of other projects use it besides IOx, including VegaFusion, ROAPI, Cube.js's preaggregation store. We're heavily using it to build Seafowl [2], an analytical database that's optimized for running SQL queries directly from the user's browser (caching, CDNs, low latency,