octosql

OctoSQL[1] does a pretty good job of allowing you to query JSON (and CSV) with SQL. [1] https://github.com/cube2222/octosql

Also worth mentioning that we support our employees working on open source during their Friday projects where they're free to do anything to grow as engineers. Many choose to do open source and we don't require them to do it under our umbrella. In fact the guy from my team who is the acting TL of the OpenTF project is the creator and maintainer of https://github.com/cube2222/octosql

Never got it to anything close to a finished state, instead moving on to doing the same prototype in llvm and then cranelift. That said, here's some of the wazero-based code on a branch - https://github.com/cube2222/octosql/tree/wasm-experiment/was... It really is just a very very basic prototype.

Just thought I'd plug Octosql[1] which I've enjoyed using for this. It parses CSV and JSON, which are the file types I parse the most. [1] https://github.com/cube2222/octosql/

I'm looking for this but can't find it, how does this project compare to differential dataflow? As a sibling commenter mentioned, it's built on timely dataflow (which is lower-level), but that already has differential dataflow[0] built on top of it by the same authors. How do they differ? (btw. I think dataflow is very cool as a computing model (not just timely dataflow), to the point of building OctoSQL[1] around it, so I'm really curious about the details here) [0]: https://github.com/TimelyDataflow/differential-dataflow [1]: https://github.com/cube2222/octosql

OctoSQL[0] or DuckDB[1] will most likely be much simpler, while going through 10 GB of JSON in a couple seconds at most. Disclaimer: author of OctoSQL [0]: https://github.com/cube2222/octosql [1]: https://duckdb.org/

This is really cool! With their Postgres scanner[0] you can now easily query multiple datasources using SQL and join between them (i.e. Postgres table with JSON file). Something I previously strived to build with OctoSQL[1]. There's even predicate push-down to the underlying databases (for Postgres)! It's amazing to see how quickly DuckDB is adding new features. Not a huge fan of C++, which is right now used for authoring extensions, it'd be really cool if somebody implemented a Rust extension SDK, or even something like Steampipe[2] does for Postgres FDWs which would provide a shim for quickly implementing non-performance-sensitive extensions for various things. Godspeed! [0]: https://duckdb.org/2022/09/30/postgres-scanner.html [1]: https://github.com/cube2222/octosql [2]: https://steampipe.io

Very interesting, sounds kinda like GraphQL counterpart to https://github.com/cube2222/octosql

Congrats on the Show HN! It's great to see more tools in this area (querying data from various sources in-place) and the Lambda use case is a really cool idea! I've recently done a bunch of benchmarking including ClickHouse Local, and the usage was straightforward, with everything working as it's supposed to. Just to comment on the performance avenue though, one area I think ClickHouse could still possibly improve on - vs OctoSQL[0] at least - is that it seems like the JSON datasource is slower, especially if only a small part of the JSON objects is used. If only a single field of many is used, OctoSQL lazily parses only that field, and skips the others, which yields non-trivial performance gains on big JSON files with small queries. Basically, for a query like `SELECT COUNT(*), AVG(overall) FROM books.json` with the Amazon Review Dataset