HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
cube2222
octosql
Go2019–2024
cube2222 / octosql

octosql

OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.

Command LineData & Databases
Explore on GitHubRead Hacker News discussions
9Distinct people
72Mentions
1Explicit recommenders
5.3KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

This looks really cool! Especially using datafusion underneath means that it probably is blazingly fast. If you like this, I recommend taking a look at OctoSQL[0], which I'm…

cube2222 used a recommendation phrase. 4 indexed comments match. This is a phrase match, not a verdict.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 10:56:20 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year72 dated mentions

Observed mentions: 2019-07-16 13:04:35 UTC to 2024-10-07 01:11:57 UTC.

05101520252019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 9 mentions; 3 submissions, 6 comments20192020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 23 mentions; 2 submissions, 21 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments20212022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 25 mentions; 1 submissions, 24 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 9 mentions; 0 submissions, 9 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2024

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC936
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC23221
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC404
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC25124
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC909
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC202

What Hacker News says

23 sources
cube2222·Aug 11, 2020
Hey everybody, one of the authors here! This release contains all we did in terms of our bachelor thesis[0]. This release adds temporal SQL, which means SQL on streams of events, like Kafka, which understands the time the events actually have happened. This is all designed to be as ergonomic as possible. We finally added Parquet support too. We're not so sure of our move to on disk storage, but for now you can kill and restart OctoSQL, it will stand up where it left off, without duplicating or losing messages. We also added telemetry in a hopefully non-privacy-invading way, to better understand what users actually use. Please let us know what you think! Feedback is greatly appreciated :) [0]: https://github.com/cube2222/octosql/blob/master/whitepapers/...
In OctoSQL Streaming: Adding Temporal SQL on Streams
cube2222·Aug 11, 2020
I do not really agree with the two cons the authors listed. Null handling isn't intuitive in the beginning, but it makes it harder to let missing data go unnoticed. The expression / table thing can be solved like we solved it in OctoSQL[0], and I think others have solved it in a similar way. Whenever you have more than a single scalar value in expression position, just create a tuple, or tuple of tuples out of it, which does act like a single value. [0]: https://github.com/cube2222/octosql
In We Can Do Better Than SQL
cube2222·Aug 5, 2020
Hey everybody, one of the authors here! For those unfamiliar with OctoSQL, it’s a local database/file query tool which lets you use SQL to query multiple databases and file formats simultaneously. This is a very big release we’ve been working on for almost a year in terms of our bachelor thesis. It’s an almost-total rewrite from the ground up. It adds support for streams of events, event time semantics, watermarks and multiple temporal SQL extensions, such as triggers, inspired by last years “One SQL To Rule Them All” paper. It also adds durability, which means you can kill and restart the OctoSQL process, there will be no record duplication and it will start where it left off. You can see the features for yourself. Following however, I’ll describe some of the stuff we learned, what we’d do differently. We’re an open source project, so no
Read the rest
bullshit marketing here, I think such “lessons learned” are very valuable overall. First and foremost, I think we hadn’t really decided what we wanted OctoSQL to be. Thus it evolved into a mutation of two ideas. The first one being a stream processing sytem like Flink or Spark. The second one being a local developer tool that’s quick, friendly, simple to use and very elastic, much akin to jq, jql or textql. I don’t think building such a mutation well is in our reach currently, so OctoSQL will have to choose one going forward. Now if we go the stream processing system path, then OctoSQL’s architecture, similar to the Volcano query engine, is much slower than if it’d use a column oriented micro-batch internal representation. If we’d create a system like this today, we’d probably orient it around Apache Arrow, like DataFusion did. We’ve also stood with our decision of connecting to actual databases only in the execution phase, which resulted in many schema-based optimisations to be impossible. However, if we go the local tool path, then everything we did around durability, building everything around local transactional Badger key-value storage, providing restartability and exactly-once semantics in the face of restarts isn’t worth much, and has a hugely negative impact on performance (10x+), which we saved a bit by parallelizing everything. Though it is quite magical when restarting starts where it left off. Where next? Maybe we’ll scrap all the transactional storage stuff, and keep octosql a handy single-machine tool to query many databases and explore data. Not trying to be something it won't ever really manage to become. The roadmap is current too, with stuff like ergonomic json manipulation to do. The lack of columnar processing is a cons, but a boon too, we can try to be as ergonomic and elastic as possible. Is the restartability feature useful to you, what do you think? We’d really love to hear your feedback about all the aforementioned issues, and how they’d impact your usage of OctoSQL! Please comment! In regards to development speed and releases, as said before, we did all this as our bachelor thesis (it's been pretty intense connected with University and a job) so work has slowed down recently as we regain our energy. But we're slowly feeling more motivated once again, so hopefully it'll gain more momentum again now. Anyways, it's been a great ride so far, hope some of you will find it useful! Our bachelor thesis is also publicly available, it contains a more detailed description of implementation details, tradeoffs and comparisons to other projects: https://github.com/cube2222/octosql/raw/master/whitepapers/t... BTW If you like what octosql is doing and would like to help speed up development (with the vision I've shown above) feel free to sponsor us on GitHub, we'll appreciate it and be able to spend more time on it.
In OctoSQL v0.3.0: Streaming, Temporal SQL on Event Streams in Go
cube2222·May 20, 2020
Sad to see this happen as I really like the idea! If you're interested in Drill, check out OctoSQL[0]. It shares the same vision of querying multiple datasources using pure SQL, and pushing down as much operations as possible to the underlying datasource. Moreover, there's been a huge rewrite under way this past year, ready to use on the master branch, yet unreleased however (will be available soon). It adds Kafka and Parquet support and most importantly first-class unbounded stream support. Including temporal SQL extensions for working with event time metadata (instead of system time) so you can use stuff like live updated time window aggregations on incoming kafka streams. It also now uses on-disk badger storage as the primary way to store its state, so you can do Group Bys / Joins with lots of keys, and restarts of OctoSQL won't alter
Read the rest
the final result (exactly-once semantics). Make sure to check it out, it's also very simple to get going locally! Disclosure: I'm one of the main contributors. [0]:https://github.com/cube2222/octosql
In Apache Drill's Future
cube2222·May 1, 2020
Haven't used DGraph itself but I've used the storage engine they built for it - Badger, an alternative to RocksDB better optimized for SSD's - in two projects. One was for event saving and retrieval, which was able to sustain a stable 60k writes /s with simultaneous 10k reads /s. It worked great overall, with stuff needing nontrivial tuning being 1. RAM usage 2. If you overwhelm it with writes it'll stall to keep up with level 0/level 1 compactions. Another one is OctoSQL[1], where we're building exactly-once event-time based stream processing all around Badger. So far it was a breeze and I don't think we'd build it if not for Badger. Overall, at least the storage engine they're using is awesome, and I can definitely recommend it! [1]:https://github.com/cube2222/octosql
In Jepsen: Dgraph 1.1.1
cube2222·Apr 13, 2020
Indeed, I remember! :) The parquet support is mostly ready as far as I know here https://github.com/cube2222/octosql/pull/153/ but I haven't tested it myself yet. However, we'll surely have it ready for the "streaming" release which is planned for May/June. Regarding the parsing part, that's interesting. Though the cockroachdb parser is a _great_ lecture about how to add nice error messages with goyacc. (You add a special error token which captures anything after a syntax error, so the parser doesn't break down. At least something around this, haven't yet dived deep into it.) Though I know Go itself uses a handwritten parser.
In Writing a SQL database from scratch in Go
cube2222·Mar 21, 2020
I'm working on an open source project - OctoSQL[1] - which allows you to query and join data from different datasources using SQL. Be it MySQL, PostgreSQL, JSON, CSV or Excel. It's now also being my Bachelor thesis. Recently we've been working hard on getting event time supporting exactly-once streaming into OctoSQL. Which means we currently (on the streaming branch) support Kafka as a datasource. Run using on-disk state (which allows grouping big datasets and is still blazingly fast thanks to Badger which we use as our underlying storage). Obviously with streaming SQL extensions, which allow you to view partial results even with standard datasources. We use a dataflow-like retraction model for that. We should have a big release out soon, so stay tuned! [1]:https://github.com/cube2222/octosql
In Ask HN: What projects are you working on now?
cube2222·Feb 25, 2020
As people always do in such threads, I’d too like to chime in with alternatives I’ve been a co-author of, as this is an area I’m very interested in. The first one, which is definitely close in spirit to this project is OctoSQL[1]. We too are trying to query all the things with SQL, including JSON, though we extended the idea to supporting more data sources and file formats, with the ability to mix and match in joins and subqueries. Curious what direction this project will take though, as we’re now heading for streaming sources (like Kafka) in pure SQL! A totally different one is jql[2] where I’ve been exploring more of a lispy continuation passing style based approach to a JSON query language, as I didn’t really feel like the current ones are ergonomic. If you like this, make sure to check them out too! (And all the others people will be
Read the rest
posting, as I think they all are fascinating! This is a big area with lots of space for innovation left.) [1]: https://github.com/cube2222/octosql [2]: https://github.com/cube2222/jql
In Show HN: Querying JSON documents using SQL-like language in Scala
cube2222·Feb 25, 2020
If you’re interested in stuff like this, check out “streaming 101: the world beyond batch”[1], it’s a great blog post series about exactly once event time streaming. And keep an eye on OctoSQL[2] too, as we’ll soon be making a big release making everything streaming friendly (with pure sql obviously!). We’ve been really inspired with articles like this one. [1]: https://www.oreilly.com/radar/the-world-beyond-batch-streami... [2]: https://github.com/cube2222/octosql
In Questioning the Lambda Architecture (2014)
cube2222·Feb 15, 2020
I really like how Flink is incorporating SQL as a first class citizen. If you're interested in stuff like this, keep an eye on OctoSQL[1]. We're working hard on event time streaming with exactly-once semantics in pure SQL, being greatly inspired by the work and research done by the Flink contributors. It's our Bachelor Thesis actually. Though we have much left to work on, with features like distributed execution, so there's still a long way ahead! [1]: https://github.com/cube2222/octosql
In Apache Flink 1.10

Showing the 15 most recent of 23 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.