datafusion

Here are a few examples you might find credible: - pola-rs/polars: https://github.com/pola-rs/polars/pull/26823 - ~2.68x median speedup of primitive-to-boolean casting credited to Claude Opus 4.6 - pydantic/monty: https://github.com/pydantic/monty/pull/643 - ~53x speedup (488ms down to 9.2ms) of bytes substring search generated with Claude Code - numpy/numpy: https://github.com/numpy/numpy/pull/31573 - ~21x speedup of Python datetime → datetime64 conversion, with Claude Code used for profiling and implementing the performance improvements - apache/datafusion: https://github.com/apache/datafusion/pull/21182 - up to 49x speedup of LIMIT queries by eliminating unnecessary sorts and pushing limits into file scans, generated with Claude Code - apache/datafusion: https://github.com/apache/datafusion/pull/21651 - ~4.39x speedup of ClickBench Q6

Co-partitioning of range-partitioned data is now supported on `main` too: see https://github.com/apache/datafusion/issues/23183 and the epic https://github.com/apache/datafusion/issues/22395

You might find https://github.com/apache/datafusion and https://github.com/datafusion-contrib/datafusion-federation of interest

Yes! This is a great idea. Thanks for mentioning it! GlueSQL could be another cool target too (https://github.com/gluesql/gluesql). I think there's a fun exploration in taking a storage engine and seeing if it's compatible with different SQL layers — GlueSQL as a simpler starting point, DataFusion as the more complete option (https://github.com/apache/datafusion). Plugging into a real SQL engine seems like a great strategy for uncovering bugs, fixing correctness issues, and discovering what use cases the storage layer actually needs to support. It's also interesting how certain SQL engines and storage engines tend to align well with each other — the right pairing can unlock a lot. Definitely a direction I want to explore.

This will obviously depend on which implementation you use. Using the rust arrow-rs crate you at least get panics when you overflow max buffer sizes. But one of my enduring annoyances with arrow is that they use signed integer types for buffer offsets and the like. I understand why it has to be that way since it's intended to be cross-language and not all languages have unsigned integer types. But it does lead to lots of very weird bugs when you are working in a native language and casting back and forth from signed to unsigned types. I spent a very frustrating day tracking down this one in particular https://github.com/apache/datafusion/issues/15967