HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
multiprocessio
dsq
Go2022–2025
multiprocessio / dsq

dsq

Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.

10Distinct people
25Mentions
1Explicit recommenders
3.9KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

Nice work! I have a similar tool built on SQLite [0] and there are a number of similar tools (compared in my readme) like octosql, q, textql, datasette, etc.…

eatonphil used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 1:56:48 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year25 dated mentions

Observed mentions: 2022-01-11 15:21:16 UTC to 2025-03-22 13:58:06 UTC.

051015202022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 19 mentions; 0 submissions, 19 comments20222023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 5 mentions; 0 submissions, 5 comments20232024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC19019
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC505
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC101

What Hacker News says

19 sources
SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions! As far as similar tools go, if you're interested, I recommend taking a look at DataFusion[0], dsq[1], and OctoSQL[2]. DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats. dsq is based on SQLite which means it has to load data into SQLite first, but then gives you the whole breath of SQLite, it also supports many data formats, but is slower at the same time. OctoSQL is faster, extensible through plugins, and supports incremental query execution, so you can i.e. calculate and display a running group by + count while tailing a log file. It also supports normal databases, not just file formats, so you can i.e. join with a Postgres table. [0]: https://github.com/apache/arrow-datafusion [
Read the rest
In Command-line data analytics
I am currently evaluating dsq and its partner desktop app DataStation. AIUI, the developer of DataStation realised that it would be useful to extract the underlying pieces into a standalone CLI, so they both support the same range of sources. dsq CLI - https://github.com/multiprocessio/dsq DataStation desktop app - https://datastation.multiprocess.io/ Two alternative CLI tools that I looked at: sq - https://sq.io/ octosql - https://github.com/cube2222/octosql Honourable mentions to: Miller - https://miller.readthedocs.io/en/latest/index.html Dasel - https://daseldocs.tomwright.me/ These don't use SQL. VisiData is also amazing for data navigation, although it requires some effort to get the model: https://www.visidata.org/
In Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
OctoSQL is an awesome project and Kuba has a lot of great experience to share from building this project I'm excited to learn from. And while building a custom database engine does allow you to do pretty quick queries, there are a few issues. First, the SQL implemented is nonstandard. As I was looking for documentation and it pointed me to `SELECT * FROM docs.functions fs`. I tried to count the number of functions but octosql crashed (a Go panic) when I ran `SELECT count(1) FROM docs.functions fs` and `SELECT count(*) FROM docs.functions fs` which is what I lazily do in standard SQL databases. (`SELECT count(fs.name) FROM docs.function fs` worked.) This kind of thing will keep happening because this project just doesn't have as much resources today as SQLite, Postgres, DuckDB, etc. It will support a limited subset of SQL. Second, the stan
Read the rest
dard library seems pretty small. When I counted the builtin functions there were only 29. Now this is an easy thing to rectify over time but just noting about the state today. And third this project only has builtin support for querying CSV and JSON files. Again this could be easy to rectify over time but just mentioning the state today. octosql is a great project but there are also different ways to do the same thing. I build dsq [0] which runs all queries through SQLite so it avoids point 1. It has access to SQLite's standard builtin functions plus a battery of extra statistic aggregation, string manipulation, url manipulation, date manipulation, hashing, and math functions custom built to help this kind of interactive querying developers commonly do [1]. And dsq supports not just CSV and JSON but parquet, excel, ODS, ORC, YAML, TSV, and Apache and nginx logs. A downside to dsq is that it is slower for large files (say over 10GB) when you only want a few columns whereas octosql does better in some of those cases. I'm hoping to improve this over time by adding a SQL filtering frontend to dsq but in all cases dsq will ultimately use SQLite as the query engine. You can find more info about similar projects in octosql's Benchmark section but I also have a comparison section in dsq [2] and an extension of the octosql benchmark with different set of tools [3] including duckdb. Everyone should check out duckdb. :) [0] https://github.com/multiprocessio/dsq [1] https://github.com/multiprocessio/go-sqlite3-stdlib [2] https://github.com/multiprocessio/dsq#comparisons [3] https://github.com/multiprocessio/dsq#benchmark*
In OctoSQL allows you to join data from different sources using SQL
This is a cool project! But if you just want a simple CLI, you can query Excel and ODS files with dsq [0]. It's the same end result without needing to carry plugins around and with support for many additional data formats. Plus a growing standard library of functions that don't come built into SQLite such as best-effort date parsing, URL parsing/extraction, statistical aggregation functions, math functions, string and regex helpers, hashing functions and so on [1]. An annoying thing about this extension-based style of file support is needing to create a new table for every new file if the schema is different. This is a limitation [2] of sqlite unfortunately. dsq doesn't work this way so it doesn't have that limit. On the other hand, if you go this route you can more easily combine with other extensions. That's not really possible with dsq
Read the rest
In Xlite: Query Excel and Open Document spreadsheets as SQLite virtual tables
Hey folks! I quit my job at Oracle almost a year ago now to build DataStation. It's an app I've wanted as an engineering manager for years. It's entirely open-source and while I've had a few awesome contributors I'm mostly the only person on it. It has been funded out of contract development and savings. DataStation helps you query a variety of data sources (conventional SQL like PostgreSQL and MySQL, non-SQL like Prometheus or Elasticsearch), files and HTTP APIs. It is not a SQL layer on top of these various APIs like FDW in Postgres or Apache Calcite. DataStation just tries to abstract away glue code. So in DataStation for Prometheus you query with PromQL. For Elasticsearch you query with Lucene. And for SQL databases you query with their SQL dialect. But you don't need to remember how to use the appropriate library for your language. Y
Read the rest
ou just need your own credentials. DataStation is made of panels (other apps might call them cells) that each produce a result. Panels can refer to other panels. These allow you to build workflows that cross the boundary of a particular datasource. For example you might have some data in a CSV a product manager gave you and the bulk of your data is in PostgreSQL. In DataStation you could pull in the CSV with a File panel and pull in the Postgres data with a Database panel. Then you can join both panel results in a Code panel using your favorite language like Python, Ruby, R, Node, Julia, etc. You can even script Code panels in a SQLite dialect with a bunch of rich addons (url parsing, best-effort date parsing, statistics aggregation, etc.): https://github.com/multiprocessio/go-sqlite3-stdlib. You can watch a simple introductory video: https://www.youtube.com/watch?v=q_jRBvbwIzU. Or if you want to see that cross-datasource interaction taken to an extreme, check out this video using Postgres metadata to filter log data in Elasticsearch to do historic request analysis on a subset of customers: https://www.youtube.com/watch?v=tIh99YVHoRE. DataStation is mainly a desktop app today where the end result is that you export graph SVGs or HTML tables or markdown tables or just a CSV file. All this data stays on your laptop so it's as easy to use in a corporate environment as any existing SQL IDE or Jupyter Notebook. In the last year it's reached 1.5k stars on Github, over 1000 unique users and currently on-average about 40 fairly active users per month (defined as having opened the app more than a few times). Since it's only just now 12 months old it's been going through a lot of maturing during this time. If you've tried it before and it was buggy or too slow it's probably worth another try now if you're still interested. DataStation is primarily an Electron app but the code that evaluates panels is written in Go. The Go evaluation code forms the backbone of another app you may have seen around HN, dsq: https://github.com/multiprocessio/dsq, which is a limited version of DataStation as a CLI for querying files with SQL. In the future I'd like to see more people using it as a server app where my goal is to support read-only dashboards and recurring exports. That part is still work-in-progress. You can find a ton of tutorials on how to interact with supported databases on the DataStation website: https://datastation.multiprocess.io/docs/. Looking forward to your feedback!
In Show HN: DataStation – App to easily query, script, and visualize data
I use Go heavily cross-platform developing DataStation [0] and dsq [1]. I am not an expert. And I don't have proof for it but on some rudimentary benchmarks the Linux-specific file idioms in the Go standard library definitely don't seem to translate well to even macOS let alone Windows. For example some good streaming techniques for reading large files on Linux that work really well there seemed to be pretty bad on macOS. I think Amos has presented more proof than I can on the topic of just how Linux-influenced Go is. And I think it is fine for the majority of Go users because the majority users of Go are building server apps or Linux CLIs. Amos has spent some time building cross-platform desktop systems with Go for itch.io and I think I'm seeing some of the same things they are in that scenario. I think this is a reasonable article. I di
Read the rest
dn't notice anything too flame-y myself but if for you Amos gets flame-y at any point I think that's worth ignoring because there does seem to be something up with Go in cross-platform applications. Amos does have good experience here. (Go look at itchi.io's github like wharf [2] or butler [3] where they are/were the main contributor.) I like Go a lot and for most things I'd keep using it still. Just sharing some observations. [0] https://github.com/multiprocessio/datastation [1] https://github.com/multiprocessio/dsq [2] https://github.com/itchio/wharf [3] https://github.com/itchio/butler
In Lies we tell ourselves to keep using Golang
In addition to jq (already mentioned) here are some other useful CLI tools for dealing with JSON data. Descriptions are directly from GitHub. Note: dsq also has a companion GUI app called DataStation[0] if you're looking for that. fx: Command-line JSON processing tool - https://github.com/antonmedv/fx dasel: JSON, YAML, TOML, XML, and CSV query and modification tool - https://github.com/TomWright/dasel dsq: CLI tool for running SQL queries against JSON, CSV, Excel, Parquet, and more - https://github.com/multiprocessio/dsq gron: Make JSON greppable - https://github.com/tomnomnom/gron jello: Filter JSON and JSON Lines data with Python syntax - https://github.com/kellyjonbrazil/jello jless: Command-line pager for JSON data - https://github.com/PaulJuliusMartinez/jless jid: Json incremental digger - https://github.com/simeji/jid jql: JSON que
Read the rest
In Ask HN: Local Tools for Viewing JSON
I've got a lot of good first projects for github.com/multiprocessio/datastation and github.com/multiprocessio/dsq. If you have any experience in Go it will be easy to get you up and running. I've already had a few students and self-learners start contributing. Join #dev on discord.multiprocess.io and say hi and I'll show you some of the stuff you can help with.
In Ask HN: Who is looking for help on a project?
SEEKING WORK | NYC | Remote OK 10 years as a dev and eng manager at startups and Oracle. Take a look at notes.eatonphil.com and github.com/eatonphil please. If you still need a resume I can send one. Email phil@multiprocess.io. I'm looking for at most 20 hours a week so I can continue working on my company, multiprocess.io. I am building github.com/multiprocessio/datastation and github.com/multiprocessio/dsq in public and they are used by a number of developers. I host a hacker Discord and a monthly virtual Meetup to expose devs to internals of compilers, databases, emulators, browsers, operating systems, etc. Besides the multiprocess work, I have a numerous public Github projects and, fun fact, 10 of my technical blog posts were on the front of HN last year. I'd be particularly excited to help you out with your internally-developed compi
Read the rest
lers/interpreters/databases. But I'm also happy to help out with business applications.
In Ask HN: Freelancer? Seeking freelancer? (March 2022)

Showing the 15 most recent of 19 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.