GYGys·Feb 28, 2023
For cross compiling a Golang application I used this alternative: https://github.com/multiprocessio/go-sqlite3-stdlib
In sqlean: A set of SQLite extensionsmultiprocessio
go-sqlite3-stdlib
Go2022–2023
multiprocessio / go-sqlite3-stdlib
go-sqlite3-stdlib
A standard library for mattn/go-sqlite3 including best-effort date parsing, url parsing, math/string functions, and stats aggregation functions
3Distinct people
8Mentions
0Explicit recommenders
129GitHub stars
What to know before exploring
What it is
A standard library for mattn/go-sqlite3 including best-effort date parsing, url parsing, math/string functions, and stats aggregation functions
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- multiprocessio/dsq · linked in 3 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 12:54:37 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by month8 dated mentions
Observed mentions: 2022-05-26 20:25:19 UTC to 2023-02-28 17:08:41 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| May 2022 | 2022-05-01 00:00:00 UTC | 2022-06-01 00:00:00 UTC | 2 | 1 | 1 |
| Jun 2022 | 2022-06-01 00:00:00 UTC | 2022-07-01 00:00:00 UTC | 2 | 0 | 2 |
| Jul 2022 | 2022-07-01 00:00:00 UTC | 2022-08-01 00:00:00 UTC | 1 | 0 | 1 |
| Aug 2022 | 2022-08-01 00:00:00 UTC | 2022-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2022 | 2022-09-01 00:00:00 UTC | 2022-10-01 00:00:00 UTC | 1 | 0 | 1 |
| Oct 2022 | 2022-10-01 00:00:00 UTC | 2022-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2022 | 2022-11-01 00:00:00 UTC | 2022-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2022 | 2022-12-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 1 | 0 | 1 |
| Jan 2023 | 2023-01-01 00:00:00 UTC | 2023-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2023 | 2023-02-01 00:00:00 UTC | 2023-03-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
8 sourcesI worked on a set of extensions for mattn/go-sqlite3, written in Go, inspired but a little further along (in functionality and test coverage) than SQLean. Useful if you want to see examples of this in Go or are using Go rather than Rust to embed SQLite.
https://github.com/multiprocessio/go-sqlite3-stdlib
In SQLite-loadable-rs: A framework for building SQLite Extensions in RustAdding user-defined functions to SQLite is not difficult, and the mechanism is quite flexible. You can create extensions and load them when you create the SQLite connection to have the functions available in queries. I wrote a blog post explaining how to do that using Rust, and the example is precisely a `regex_extract` function [0].
If you need them, you also have a "stdlib" implemented for Go [1] and a pretty extensive collection of extensions [2]
[0]: https://ricardoanderegg.com/posts/extending-sqlite-with-rust...
[1]: https://github.com/multiprocessio/go-sqlite3-stdlib
[2]: https://github.com/nalgeon/sqlean
In SQLite: Past, Present, and FutureOctoSQL is an awesome project and Kuba has a lot of great experience to share from building this project I'm excited to learn from.
And while building a custom database engine does allow you to do pretty quick queries, there are a few issues.
First, the SQL implemented is nonstandard. As I was looking for documentation and it pointed me to `SELECT * FROM docs.functions fs`. I tried to count the number of functions but octosql crashed (a Go panic) when I ran `SELECT count(1) FROM docs.functions fs` and `SELECT count(*) FROM docs.functions fs` which is what I lazily do in standard SQL databases. (`SELECT count(fs.name) FROM docs.function fs` worked.)
This kind of thing will keep happening because this project just doesn't have as much resources today as SQLite, Postgres, DuckDB, etc. It will support a limited subset of SQL.
Second, the stan
Read the rest
dard library seems pretty small. When I counted the builtin functions there were only 29. Now this is an easy thing to rectify over time but just noting about the state today.
And third this project only has builtin support for querying CSV and JSON files. Again this could be easy to rectify over time but just mentioning the state today.
octosql is a great project but there are also different ways to do the same thing.
I build dsq [0] which runs all queries through SQLite so it avoids point 1. It has access to SQLite's standard builtin functions plus a battery of extra statistic aggregation, string manipulation, url manipulation, date manipulation, hashing, and math functions custom built to help this kind of interactive querying developers commonly do [1].
And dsq supports not just CSV and JSON but parquet, excel, ODS, ORC, YAML, TSV, and Apache and nginx logs.
A downside to dsq is that it is slower for large files (say over 10GB) when you only want a few columns whereas octosql does better in some of those cases. I'm hoping to improve this over time by adding a SQL filtering frontend to dsq but in all cases dsq will ultimately use SQLite as the query engine.
You can find more info about similar projects in octosql's Benchmark section but I also have a comparison section in dsq [2] and an extension of the octosql benchmark with different set of tools [3] including duckdb.
Everyone should check out duckdb. :)
[0] https://github.com/multiprocessio/dsq
[1] https://github.com/multiprocessio/go-sqlite3-stdlib
[2] https://github.com/multiprocessio/dsq#comparisons
[3] https://github.com/multiprocessio/dsq#benchmark*
This is a cool project! But if you just want a simple CLI, you can query Excel and ODS files with dsq [0]. It's the same end result without needing to carry plugins around and with support for many additional data formats. Plus a growing standard library of functions that don't come built into SQLite such as best-effort date parsing, URL parsing/extraction, statistical aggregation functions, math functions, string and regex helpers, hashing functions and so on [1].
An annoying thing about this extension-based style of file support is needing to create a new table for every new file if the schema is different. This is a limitation [2] of sqlite unfortunately. dsq doesn't work this way so it doesn't have that limit.
On the other hand, if you go this route you can more easily combine with other extensions. That's not really possible with dsq
Read the rest
Hey Kuba!
> Especially on the community building aspect, it's really impressive that you've been able to spark so many communities on various platforms (Reddit, GitHub, Discord, etc.)!
Yeah it's been so cool to see so many people come together, hobbyists and professionals.
> On a more technical note, since dsq is based on the "load it into SQLite and query it from there" architecture, have you considered integrating with the plugin ecosystems of other existing projects based on that same architecture, like Datasette[0]? It seems like a way to add a lot of value to your tools without much work.
Interesting idea! I haven't looked into Datasette too much. And I haven't thought about plugins too much either. The most I've done is extend the SQLite standard library [0] and I hope to continue growing that. I'd be curious to hear what specifical
Read the rest
ly people like from Datasette they'd like to see in dsq.
> On a more commercial note, overall I think tools like this are very hard to monetize, because right now they're just a fairly niche use case, between - as you mentioned - full blown data analytics platforms and observability query systems, as well as standard unix tools. Especially since if you need the analytics a lot, you'll probably have time to integrate it into your preferred analytics solution (like BigQuery). Do you have any thoughts on that?
My idea was always to focus on smaller and less mature organizations, probably ones that have been around for 10+ years. They aren't using BigQuery, they prefer to host everything themselves, and they don't yet realize there are tools like DataStation that they can easily run to make analytics easier.
I've worked at a bunch of companies like this so I know the market exists. Actually I have been surprised how many people outside of this market showed up in the DataStation community. I've seen Googlers, MS-ers, modern startups, data science teams show up interested in DataStation compared to what they're already using.
For me it's just been a matter of time (and funding) to build out the product to serve these communities commercially as a SaaS or enterprise product.
[0] https://github.com/multiprocessio/go-sqlite3-stdlib
Hey folks! I quit my job at Oracle almost a year ago now to build DataStation. It's an app I've wanted as an engineering manager for years. It's entirely open-source and while I've had a few awesome contributors I'm mostly the only person on it. It has been funded out of contract development and savings.
DataStation helps you query a variety of data sources (conventional SQL like PostgreSQL and MySQL, non-SQL like Prometheus or Elasticsearch), files and HTTP APIs. It is not a SQL layer on top of these various APIs like FDW in Postgres or Apache Calcite.
DataStation just tries to abstract away glue code. So in DataStation for Prometheus you query with PromQL. For Elasticsearch you query with Lucene. And for SQL databases you query with their SQL dialect. But you don't need to remember how to use the appropriate library for your language. Y
Read the rest
ou just need your own credentials.
DataStation is made of panels (other apps might call them cells) that each produce a result. Panels can refer to other panels. These allow you to build workflows that cross the boundary of a particular datasource. For example you might have some data in a CSV a product manager gave you and the bulk of your data is in PostgreSQL. In DataStation you could pull in the CSV with a File panel and pull in the Postgres data with a Database panel. Then you can join both panel results in a Code panel using your favorite language like Python, Ruby, R, Node, Julia, etc. You can even script Code panels in a SQLite dialect with a bunch of rich addons (url parsing, best-effort date parsing, statistics aggregation, etc.): https://github.com/multiprocessio/go-sqlite3-stdlib.
You can watch a simple introductory video: https://www.youtube.com/watch?v=q_jRBvbwIzU. Or if you want to see that cross-datasource interaction taken to an extreme, check out this video using Postgres metadata to filter log data in Elasticsearch to do historic request analysis on a subset of customers: https://www.youtube.com/watch?v=tIh99YVHoRE.
DataStation is mainly a desktop app today where the end result is that you export graph SVGs or HTML tables or markdown tables or just a CSV file. All this data stays on your laptop so it's as easy to use in a corporate environment as any existing SQL IDE or Jupyter Notebook.
In the last year it's reached 1.5k stars on Github, over 1000 unique users and currently on-average about 40 fairly active users per month (defined as having opened the app more than a few times).
Since it's only just now 12 months old it's been going through a lot of maturing during this time. If you've tried it before and it was buggy or too slow it's probably worth another try now if you're still interested.
DataStation is primarily an Electron app but the code that evaluates panels is written in Go. The Go evaluation code forms the backbone of another app you may have seen around HN, dsq: https://github.com/multiprocessio/dsq, which is a limited version of DataStation as a CLI for querying files with SQL.
In the future I'd like to see more people using it as a server app where my goal is to support read-only dashboards and recurring exports. That part is still work-in-progress.
You can find a ton of tutorials on how to interact with supported databases on the DataStation website: https://datastation.multiprocess.io/docs/.
Looking forward to your feedback!
Show HN: A standard library for mattn/go-sqlite3
3 points · Open discussion