quokka

An awesome read! Something related that I found out about from HN a few months back is another engine called quokka. It's particularly interesting and applicable how quokka schedules distributed queries to outperform Spark https://github.com/marsupialtail/quokka/blob/master/blog/why...

Hi Justin, you might be interested in my blog: https://github.com/marsupialtail/quokka/blob/master/blog/bac... advocating a cloud based approach. You don't have to use the system I am building, but it's worth thinking about that design.

SQL support is very challenging. I work on Quokka (https://github.com/marsupialtail/quokka). I support Iceberg reads. Recently we are adding SQL support from just parsing the DuckDB logical plan, though that is very challenging as well. The Python world lacks a standard for a plug and play SQL query optimizer. Apache Calcite is good for the JVM world, but not great if you are trying to cut out the JVM.

Hi -- I am the author of Quokka: https://github.com/marsupialtail/quokka, trying to be distributed Polars. I am trying to go for API compatibility, or at least supporting most of the API. I am not focused on complex data types though.

In case people are interested, I wrote a post about fault tolerance strategies of data systems like Spark and Flink: https://github.com/marsupialtail/quokka/blob/master/blog/fau... The key difference here is that these systems don't store data, so fault tolerance means recovering within a query instead of not losing data.

would love to collaborate on an integration with pyquokka (https://github.com/marsupialtail/quokka) once I put out a stable release end of this month :-)

I saw your engine a week or so ago. Good stuff. I wanted to add it to dbdb.io but I couldn't find an SVG version of this: https://github.com/marsupialtail/quokka/blob/master/docs/doc... Can you email it to me?

I am writing a Python based SQL query engine: https://github.com/marsupialtail/quokka. My personal goal is to get Andy Pavlo to mention it in his year-end blogs. I agree with many of the points made in the blog by Andy. Writing a distributed database has become way easier due to open source components like Ray, Arrow, Velox, DuckDB, SQLGlot etc. I personally believe we will see a switch from JVM based technologies to Rust/C based with Python wrapper

Last year around this time I was joking with people if you can hack together something like SparkSQL with all the best Python open-source: Polars/Arrow/Ray/SQLGlot. So here it is. Definitely NOT production ready -- but should work for most advertised use cases. Only has a dataframe/datastream API right now, working on full SQL support. Hard because of all the weird SQL people write :-P