datafusion

BTW here is a fun exercise that takes this idea to the extreme. Who can build a custom file format that gets the best ClickHouse performance (on DataFusion): https://github.com/apache/datafusion/issues/13448 Disclaimer I am on the PMC of Apache DataFusion, so am totally a fan boy.

Those are interesting results! I'm not surprised it works a lot better for metrics than logs and traces. Something I'd really love to have for logs/traces processing is the ability to query clp[1][2] with a dataframe interface (e.g. datafusion [3]). While I'm on that subject, I'd also prefer that interface for metrics processing. I don't need real-time streaming metrics graphs, it's perfectly fine to compute one on-demand. I suspect something like clp is the way to go for logs-like data, that is, low entropy text with a lot of numerical content. [1] https://www.uber.com/blog/reducing-logging-cost-by-two-order... [2] https://www.uber.com/blog/modernizing-logging-with-clp-ii/ [3] https://github.com/apache/datafusion

I really like the look of vortex[1]! One of my industry pet peeves is all the useless utf-8 server log bytes. I'd like to log data in a sane, schemaful, binary format and this looks like it could be a good way to do that. Bonus points if we can wire this up as a physical layer for e.g. datafusion[2] so I can analyze my logs with the dataframe abstraction. EDIT: Question about FSST--lets say I build a strings table like: struct Strings { compressor: fsst::Compressor, compressed: Vec<Vec<u8>> } Is there some optimal length for compressed given the 255 symbols limit? [1] https://github.com/spiraldb/vortex [2] https://github.com/apache/datafusion

duckdb is mit licensed. [1] datafusion is apache v2 licensed. [2] pg_lakehouse built on top of data fusion is AGPL v3 + business licensed. [3] [1] https://github.com/duckdb/duckdb/blob/main/LICENSE [2] https://github.com/apache/datafusion/blob/main/LICENSE.txt [3] https://github.com/paradedb/paradedb/blob/dev/LICENSE Most companies won't touch AGPL v3 license. Maybe not a bad thing, but FYI.

Python's Substrait seems like the biggest/most-used competitor-ish out there. I'd love some compare & contrast; my sense is that Substrait has a smaller ambition, more wants to be a language for talking about execution rather than a full on optimization/execution engine. https://github.com/substrait-io/substrait . (Edit: ah, there's a recent talk discussing PyVelox trying to get Substrait integration. https://www.youtube.com/watch?v=l_kHxkGkNRg#t=18m22s . However there's also discussion about the un-maintainedness of some of the current Substrait work here; unclear status. https://github.com/facebookincubator/velox/issues/8895) We can also see from the Apache Arrow DataFusion discussion that they too see themselves as a bit of a Velox competitor. https://github.com/apache/arrow-datafusion/discussions/6441 It's cool to see this space matur

Agree, substrait is a really cool project! Related: if you like substrait you might want to check out datafusion too. The project is a query execution engine built on top of Apache Arrow (incl. SQL parser, query planner & optimizer, execution engine, extensible user defined functions, among others) and it implements a substrait provider and consumer: https://github.com/apache/arrow-datafusion/tree/main/datafus...

Specifically: https://github.com/apache/arrow-datafusion/tree/main/datafus...