NBnbk_2000·Oct 7, 2024
Octosql does this as well as a few other formats. I've found it useful several times.
https://github.com/cube2222/octosql
In Sq.io: jq for databases and morecube2222
octosql
Go2019–2024
cube2222 / octosql
octosql
OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
9Distinct people
72Mentions
1Explicit recommenders
5.3KGitHub stars
What to know before exploring
What it is
OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
This looks really cool! Especially using datafusion underneath means that it probably is blazingly fast. If you like this, I recommend taking a look at OctoSQL[0], which I'm…
cube2222 used a recommendation phrase. 4 indexed comments match. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- cube2222/jql · linked in 3 of the same discussions
- apache/datafusion · linked in 2 of the same discussions
- multiprocessio/dsq · linked in 2 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 10:02:05 AM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year72 dated mentions
Observed mentions: 2019-07-16 13:04:35 UTC to 2024-10-07 01:11:57 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 9 | 3 | 6 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 23 | 2 | 21 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 4 | 0 | 4 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 25 | 1 | 24 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 9 | 0 | 9 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 2 | 0 | 2 |
What Hacker News says
2 sourcesThis looks extremely cool. This is basically incremental view maintenance in databases, a problem that almost everybody (I think) has when using SQL databases and wanting to do some derived views for more performant access patterns. Importantly, they seem to support a wide breath of SQL operators, support spilling computation state to disk, and it's open-source! Interestingly, it compiles queries to Rust, so an approach similar to Redshift (which compiles queries to C++ programs).
There's already a bunch of tools in this area:
1. Materialize[0], which afaik is more big-data oriented, and doesn't pipe the results back to your database, instead storing results in S3 and serving them.
2. Epsio[1], which I've never used, seems to be very similar to this product, but is closed-source only.
3. When building OctoSQL[2], this capability was also
Read the rest
important to me and it was designed from ground up to support it. Though in practice in a tool like OctoSQL it's pretty useless (was a fun problem to solve though).
There's some things I'm curious about:
- Does it handle queries that involve complex combinations of ordering with limits in subqueries? If due to a change in an underlying table a top-n row is added, resulting in moving other rows around (and removing the current n'th) will the subsequent query parts behave as though the order was maintained when computing it, or will it fall apart (imagine a select with limit from a select with bigger limit)?
- Is it internally consistent[3]? They say it's "strongly consistent" and "It also guarantees that the state of the views always corresponds to what you'd get if you ran the queries in a batch system for the same input." so I think the answer is yes, but this one's really important.
Either way, will have to play with this, and dig into the paper (the link in the repo doesn't work, here's an arXiv link[4]). Wishing the creators good luck, this looks great!
[0]: https://materialize.com
[1]: https://www.epsio.io
[2]: https://github.com/cube2222/octosql
[3]: https://www.scattered-thoughts.net/writing/internal-consiste...
[4]: https://arxiv.org/pdf/2203.16684