HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
harelba
q
Python2012–2023
harelba / q

q

q - Run SQL directly on delimited files and multi-file sqlite databases

24Distinct people
26Mentions
0Explicit recommenders
10.4KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

q - Run SQL directly on delimited files and multi-file sqlite databases

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year26 dated mentions

Observed mentions: 2012-02-13 12:40:24 UTC to 2023-03-25 04:53:14 UTC.

02462012-01-01 00:00:00 UTC to 2013-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments20122013-01-01 00:00:00 UTC to 2014-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2014-01-01 00:00:00 UTC to 2015-01-01 00:00:00 UTC (exclusive): 3 mentions; 2 submissions, 1 comments2015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 4 mentions; 2 submissions, 2 comments2016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments20172018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 6 mentions; 1 submissions, 5 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2023

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20122012-01-01 00:00:00 UTC2013-01-01 00:00:00 UTC110
20132013-01-01 00:00:00 UTC2014-01-01 00:00:00 UTC110
20142014-01-01 00:00:00 UTC2015-01-01 00:00:00 UTC321
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC422
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC000
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC000
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC303
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC101
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC000
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC615
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC404
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC303

From the project README

An excerpt from the repository, fetched Nov 13, 2026.

Read the project’s introduction
q - Text as Data q's purpose is to bring SQL expressive power to the Linux command line and to provide easy access to text as actual data. q allows the following: Performing SQL-like statements directly on tabular text data, auto-caching the data in order to accelerate additional querying on the same file. Performing SQL statements directly on multi-file sqlite3 databases, without having to merge them or load them into memory The following table shows the impact of using caching: Rows Columns File Size Query time without caching Query time with caching Speed Improvement 5,000,000 100 4.8GB 4 minutes, 47 seconds 1.92 seconds x149 1,000,000 100 983MB 50.9 seconds 0.461 seconds x110 1,000,000 50 477MB 27.1 seconds 0.272 seconds x99 100,000 100 99MB 5.2 seconds 0.141 seconds x36 100,000 50 48MB 2.7 seconds 0.105 seconds x25 Notice that for the current version, caching is not enabled by default, since the caches take disk space. Use -C readwrite or -C read to enable it for a query, or add caching_mode to .qrc to set a new default. q's web site is https://harelba.github.io/q/ or https://q.textasdata.wiki It contains everything you need to download and use q immediately. Usage Examples q treats ordinary files as database tables, and supports all SQL constructs, such as WHERE, GROUP BY, JOINs, etc. It supports automatic column name and type detection, and provides full support for multiple character encodings. Here are some example commands to get the idea: $ q "SELECT COUNT(*) FROM ./clicks_file.csv WHERE c3 > 32.3" $ ps -ef | q -H "SELECT UID, COUNT(*) cnt FROM - GROUP BY UID ORDER BY cnt DESC LIMIT 3" $ q "select count(*) from some_db.sqlite3:::albums a left join another_db.sqlite3:::tracks t on (a.album_id = t.album_id)" Detailed examples are in here Installation. New Major Version 3.1.6 is out with a lot of significant additions. Instructions for all OSs are here. The previous version 2.0.19 Can still be downloaded from here Contact Any feedback/suggestions/complaints regarding this tool would be much appreciated. Contributions are most welcome as well, of course. Linkedin: Harel Ben Attia Twitter @harelba Email harelba@gmail.com q on twitter: #qtextasdata Patreon: harelba - All the money received is donated to the Center for the Prevention and Treatment of Domestic Violence in my hometown - Ramla, Israel.
Read the full README on GitHub

What Hacker News says

26 sources
Everyone is out here making fancy AI projects and talking about high tech stuff. In the mean time, I wrote this plain bash script to convert .ics files (a format from 1998) to CSV files. This lets you explore your calendar data without needing to setup API access and deal with authorization to your calendar software's API. (common ones all have .ics exports readily available) Combine this with CSV tooling and you got yourself some quick and easy calendar data analysis / exploration setup. Hope you like it! CSV utilities (still haven't pick a favorite one...): https://github.com/harelba/q https://github.com/BurntSushi/xsv https://github.com/wireservice/csvkit https://github.com/johnkerl/miller
In icsp – Command-line iCalendar (.ics) to CSV parser
Second this recommendation for the terminal. For my CLI toolbox, VisiData is my favorite. I find VisiData is great for quickly exploring and querying data from the CLI. It can handle many types of files (SQLite, CSV, TSV, Excel, JSON, YAML, etc). Visidata loads all the data into memory, and so is very responsive when exploring the data. It allows you to quickly do all sorts of of adhoc queries interactively, without having to write a valid SQL query. I haven't used Q. When I first heard of it, I liked the idea that Q allowed you to run random queries on CSV and TSV files. However, it seemed like it would be slow if you wanted to do follow up queries, since it had to repopulate the in memory SQLite file for each query. Though it looks like the latest version has a way to cache the generated sqlite file. So that seems like it could
Read the rest
help. Also, if I have some CSV, TSV, JSONL data sqlite-utils is useful for converting them to SQLite, and then exploring with Visidata or SQL queries. Q: https://github.com/harelba/q sqlite-utils: https://github.com/simonw/sqlite-utils
In A fast SQLite PWA notebook for CSV files
Great to see SQL being used more for such use cases! (I really like SQL) This looks nice, especially with the charting! It's good to know though that there are a few similar open source tools already: - q[0] - SQL on CSV and TSV - TextQL[1] - SQL on CSV and TSV - Datasette[2] - Running SQL on various datasets in various formats, by (if I understand it correctly) importing it to sqlite and running the query there. This one's the closest one to the posted project I think, with interactive data exploration and publishing built in. - OctoSQL[3] - SQL on files like CSV, JSON as well as non-file databases, endless streams with temporal SQL extensions and joining everything together in a single query. The performance numbers of this tool however are impressive if they will check out with real data. 1GB in 10 seconds is fast. However, looking at
Read the rest
the documentation it looks like most of the heavy lifting is done by SQLite under the hood. From the docs: > Under the hood, Superintendent loads CSVs into a SQLite database. Disclaimer: I'm the author of OctoSQL. Still actively developing a rewrite on the redesign branch. [0]: https://github.com/harelba/q [1]: https://github.com/dinedal/textql [2]: https://datasette.io [3]: https://github.com/cube2222/octosql
In Show HN: Work with CSV files using SQL. For data scientists and engineers

Showing the 15 most recent of 26 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.