datafusion

Here are a few examples you might find credible: - pola-rs/polars: https://github.com/pola-rs/polars/pull/26823 - ~2.68x median speedup of primitive-to-boolean casting credited to Claude Opus 4.6 - pydantic/monty: https://github.com/pydantic/monty/pull/643 - ~53x speedup (488ms down to 9.2ms) of bytes substring search generated with Claude Code - numpy/numpy: https://github.com/numpy/numpy/pull/31573 - ~21x speedup of Python datetime → datetime64 conversion, with Claude Code used for profiling and implementing the performance improvements - apache/datafusion: https://github.com/apache/datafusion/pull/21182 - up to 49x speedup of LIMIT queries by eliminating unnecessary sorts and pushing limits into file scans, generated with Claude Code - apache/datafusion: https://github.com/apache/datafusion/pull/21651 - ~4.39x speedup of ClickBench Q6

Co-partitioning of range-partitioned data is now supported on `main` too: see https://github.com/apache/datafusion/issues/23183 and the epic https://github.com/apache/datafusion/issues/22395

You might find https://github.com/apache/datafusion and https://github.com/datafusion-contrib/datafusion-federation of interest

Yes! This is a great idea. Thanks for mentioning it! GlueSQL could be another cool target too (https://github.com/gluesql/gluesql). I think there's a fun exploration in taking a storage engine and seeing if it's compatible with different SQL layers — GlueSQL as a simpler starting point, DataFusion as the more complete option (https://github.com/apache/datafusion). Plugging into a real SQL engine seems like a great strategy for uncovering bugs, fixing correctness issues, and discovering what use cases the storage layer actually needs to support. It's also interesting how certain SQL engines and storage engines tend to align well with each other — the right pairing can unlock a lot. Definitely a direction I want to explore.

This will obviously depend on which implementation you use. Using the rust arrow-rs crate you at least get panics when you overflow max buffer sizes. But one of my enduring annoyances with arrow is that they use signed integer types for buffer offsets and the like. I understand why it has to be that way since it's intended to be cross-language and not all languages have unsigned integer types. But it does lead to lots of very weird bugs when you are working in a native language and casting back and forth from signed to unsigned types. I spent a very frustrating day tracking down this one in particular https://github.com/apache/datafusion/issues/15967

We are actively working on supporting extension types. The mechanism is likely to be using the Arrow extension type mechanism (a logical annotation on top of existing Arrow types https://arrow.apache.org/docs/format/Columnar.html#format-me...) I expect this to be used to support Variant https://github.com/apache/datafusion/issues/16116 and geometry types (note I am an author)

I'm not sure if this is what you're looking for, but there is a proposal in DataFusion to allow user defined types. https://github.com/apache/datafusion/issues/12644

Yes, they are similar. ArkFlow is mainly based on DataFusion. Bento actually comes from Benthos. Currently, the ArkFlow project is in the early stages and no performance comparison test has been conducted, but I believe that ArkFlow will outperform them in the long run. Benthos: https://github.com/redpanda-data/benthos DataFusion: https://github.com/apache/datafusion

Location: India Remote: Yes Willing to relocate: Yes Technologies: Rust, Nodejs, Javascript, Typescript, Golang Résumé/CV: https://drive.google.com/drive/folders/1ecTn700lcmt8cqlnBTtm... Email: jatin6972@gmail.com Github: https://github.com/jatin510 Info: Hi, I'm Jagdish Parihar! A Backend Engineer with 4+ years of experience building scalable systems and microservices using Rust, Node.js, and Golang. I've contributed to open-source projects like Apache DataFusion and thrive on solving complex backend challenges. I'm exploring the opportunity to be working in the DB based startups. I am looking for an entry to be an engineer who will work on databases. Currently, I am contributing to open source, looking for part-time/full-time working with databases. Datafusion contributions: https://github.com/apache/datafusion/pulls?q=is%3Apr+author%..

BTW here is a fun exercise that takes this idea to the extreme. Who can build a custom file format that gets the best ClickHouse performance (on DataFusion): https://github.com/apache/datafusion/issues/13448 Disclaimer I am on the PMC of Apache DataFusion, so am totally a fan boy.

Those are interesting results! I'm not surprised it works a lot better for metrics than logs and traces. Something I'd really love to have for logs/traces processing is the ability to query clp[1][2] with a dataframe interface (e.g. datafusion [3]). While I'm on that subject, I'd also prefer that interface for metrics processing. I don't need real-time streaming metrics graphs, it's perfectly fine to compute one on-demand. I suspect something like clp is the way to go for logs-like data, that is, low entropy text with a lot of numerical content. [1] https://www.uber.com/blog/reducing-logging-cost-by-two-order... [2] https://www.uber.com/blog/modernizing-logging-with-clp-ii/ [3] https://github.com/apache/datafusion

I really like the look of vortex[1]! One of my industry pet peeves is all the useless utf-8 server log bytes. I'd like to log data in a sane, schemaful, binary format and this looks like it could be a good way to do that. Bonus points if we can wire this up as a physical layer for e.g. datafusion[2] so I can analyze my logs with the dataframe abstraction. EDIT: Question about FSST--lets say I build a strings table like: struct Strings { compressor: fsst::Compressor, compressed: Vec<Vec<u8>> } Is there some optimal length for compressed given the 255 symbols limit? [1] https://github.com/spiraldb/vortex [2] https://github.com/apache/datafusion

duckdb is mit licensed. [1] datafusion is apache v2 licensed. [2] pg_lakehouse built on top of data fusion is AGPL v3 + business licensed. [3] [1] https://github.com/duckdb/duckdb/blob/main/LICENSE [2] https://github.com/apache/datafusion/blob/main/LICENSE.txt [3] https://github.com/paradedb/paradedb/blob/dev/LICENSE Most companies won't touch AGPL v3 license. Maybe not a bad thing, but FYI.