datafusion

Thanks! It's using Datafusion as the query engine: https://github.com/apache/arrow-datafusion

What Andy giveth, Bill taketh away.[0] I'm more than a little annoyed that so much data engineering is still done in Scala Spark or PySpark. Both suffer from pretty high memory overhead, which leads to suboptimal resource utilization. I've worked with a few different systems that compile their queries into C/C++ (which is transparent to the developer). Those tend to be significantly faster or can use fewer nodes to process. I get that quick & dirty scripts for exploration don't need to be super optimized, and that throwing more hardware at the problem _can_ be cheaper than engineering time, but in my experience, the latter ends up costing my org tens of millions of dollars annually -- just write some code and allocate a ton of resources to make it work in a reasonable amount of time. I'm hopeful that Ballista[1], for example, will see upt

Disclosure: I am a contributor to Datafusion. I have done a lot of work in the ETL space in Apache Spark to build Arc (https://arc.tripl.ai/) and have ported a lot of the basic functionality of Arc to Datafusion as a proof-of-concept. The appeal to me of the Apache Spark and Datafusion engines is the ability to a) seperate compute and storage b) express transformation logic in SQL. Performance: From those early experiments Datafusion would frequently finish processing an entire job _before_ the SparkContext could be started - even on a local Spark instance. Obviously this is at smaller data sizes but in my experience a lot of ETL is about repeatable processes not necessarily huge datasets. Compatibility: Those experiments were done a few months ago and the SQL compatibility of the Datafusion engine has improved extremely rapidly (WINDOW f

Are there any good resources on using DataFusion in Python beyond the README [1]? [1] https://github.com/apache/arrow-datafusion/tree/master/pytho...

There is experimental support for distributed query execution with spill-to-disk between stages to support larger than memory datasets. This is implemented in the Ballista crate, which extends DataFusion. https://github.com/apache/arrow-datafusion/tree/master/balli...

OK: I have found: https://github.com/apache/arrow-datafusion "DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.DataFusion supports both an SQL and a DataFrame API for building logical query plans as well as a query optimizer and execution engine capable of parallel execution against partitioned data sources (CSV and Parquet) using threads. DataFusion also supports distributed query execution via the Ballista crate." "Use Cases: DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API."

I almost mixed this up with Apache Arrow DataFusion for a second: https://github.com/apache/arrow-datafusion

For sure, I didn't mean to imply they weren't looking at compute too! https://github.com/apache/arrow-datafusion is another example of the shared compute vision. What I was trying to point out is that (at least for Arrow core) they seem to eschew FFI and generating shared libraries in favour of from scratch implementations in other compiled languages and direct bindings in interpreted ones.

Just a heads up - ballista has been donated to apache arrow, but they also broke the Rust arrow libraries into separate repos [0][1] in a break from the mono-repo model. Read more at the announcement[3] or the merge request[4]. [0] https://github.com/apache/arrow-datafusion [1] https://github.com/apache/arrow-rs [2] https://arrow.apache.org/blog/2021/05/04/rust-dev-workflow/ [3] https://github.com/apache/arrow/pull/10096