HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
dcmoura
spyql
Jupyter Notebook2022–2023
dcmoura / spyql

spyql

Query data on the command line with SQL-like SELECTs powered by Python expressions

Command LineData & Databases
Explore on GitHubRead Hacker News discussions
5Distinct people
13Mentions
0Explicit recommenders
926GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Query data on the command line with SQL-like SELECTs powered by Python expressions

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 10:49:33 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month13 dated mentions

Observed mentions: 2022-01-24 12:35:01 UTC to 2023-06-03 15:31:59 UTC.

0123452022-01-01 00:00:00 UTC to 2022-02-01 00:00:00 UTC (exclusive): 5 mentions; 3 submissions, 2 commentsJan 20222022-02-01 00:00:00 UTC to 2022-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-03-01 00:00:00 UTC to 2022-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-04-01 00:00:00 UTC to 2022-05-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2022-05-01 00:00:00 UTC to 2022-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-06-01 00:00:00 UTC to 2022-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-07-01 00:00:00 UTC to 2022-08-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2022-08-01 00:00:00 UTC to 2022-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-09-01 00:00:00 UTC to 2022-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsSep 20222022-10-01 00:00:00 UTC to 2022-11-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2022-11-01 00:00:00 UTC to 2022-12-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2022-12-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-01-01 00:00:00 UTC to 2023-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-02-01 00:00:00 UTC to 2023-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-03-01 00:00:00 UTC to 2023-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-01 00:00:00 UTC to 2023-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-05-01 00:00:00 UTC to 2023-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-06-01 00:00:00 UTC to 2023-07-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsJun 2023

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Jan 20222022-01-01 00:00:00 UTC2022-02-01 00:00:00 UTC532
Feb 20222022-02-01 00:00:00 UTC2022-03-01 00:00:00 UTC000
Mar 20222022-03-01 00:00:00 UTC2022-04-01 00:00:00 UTC000
Apr 20222022-04-01 00:00:00 UTC2022-05-01 00:00:00 UTC211
May 20222022-05-01 00:00:00 UTC2022-06-01 00:00:00 UTC000
Jun 20222022-06-01 00:00:00 UTC2022-07-01 00:00:00 UTC000
Jul 20222022-07-01 00:00:00 UTC2022-08-01 00:00:00 UTC101
Aug 20222022-08-01 00:00:00 UTC2022-09-01 00:00:00 UTC000
Sep 20222022-09-01 00:00:00 UTC2022-10-01 00:00:00 UTC000
Oct 20222022-10-01 00:00:00 UTC2022-11-01 00:00:00 UTC211
Nov 20222022-11-01 00:00:00 UTC2022-12-01 00:00:00 UTC202
Dec 20222022-12-01 00:00:00 UTC2023-01-01 00:00:00 UTC000
Jan 20232023-01-01 00:00:00 UTC2023-02-01 00:00:00 UTC000
Feb 20232023-02-01 00:00:00 UTC2023-03-01 00:00:00 UTC000
Mar 20232023-03-01 00:00:00 UTC2023-04-01 00:00:00 UTC000
Apr 20232023-04-01 00:00:00 UTC2023-05-01 00:00:00 UTC000
May 20232023-05-01 00:00:00 UTC2023-06-01 00:00:00 UTC000
Jun 20232023-06-01 00:00:00 UTC2023-07-01 00:00:00 UTC101

What Hacker News says

5 sources
dmoura·Jan 25, 2022Submission

Show HN: SPyQL – SQL with Python in the middle

SPyQL (https://github.com/dcmoura/spyql) is SQL with Python in the middle, an open-source project fully written in Python for making command-line data processing more intuitive, readable and powerful. Try mixing in the same pot: a SQL SELECT for providing the structure, Python expressions for defining transformations and conditions, the essence of awk as a data-processing language, and the JSON handling capabilities of jq. How does a SPyQL query looks like? $ spyql “ IMPORT pendulum AS p SELECT (p.now() - p.from_timestamp(purchase_ts)).in_days() AS days_ago, sum_agg(price * quantity) AS total FROM csv WHERE department.upper() == 'IT' and purchase_ts is not Null GROUP BY 1 ORDER BY 1 TO json” < my_purchases.csv In a single statement we are 1) reading a CSV (of purchases) with automatic header
Read the rest
detection, dialect detection, type inference and casting, 2) filtering out records that do not belong to the IT department or do not have a purchase timestamp 3) summing the total purchases and grouping by how many days ago they happened, 4) sorting from the most to the least recent day and 5) writing the result in JSON format. All this without loading the full dataset into memory. The Readme is loaded with recipes and there is also a demo video: https://vimeo.com/danielcmoura/spyqldemo Any feedback is welcomed! Thank you.
105 points · Open discussion
dmoura·Jan 25, 2022
I am writing a language and CLI that mixes SQL with Python (spyql: https://github.com/dcmoura/spyql), and it is interesting to see that we are tackling some of the same problems: - code/formula reutilization where we need to repeat logic over the query - functionalities like `EXCEPT` and `REPLACE` modifiers for `SELECT *`, like in google bigquery (in most SQL databases its frustrating when you have a large number of columns and you only want to hide or replace a couple of them) I do think SQL is all over the place, and while not perfect, it’s familiar and we got used to express our queries the SQL way. At the end, since you are generating SQL to interact with databases, you would have to understand SQL in order to optimise your queries (it might be challenging to get the perfect SQL query from PRQL as you do not know statistics about the
Read the rest
tables or which indexes are available). With SPyQL I am taking a different approach that tries to extend simple SQL SELECT statements so that some of these annoying features are tackled. In addition, by using Python to define expressions and conditions you solve another problem typically present on databases: extensibility. By including an IMPORT clause in your query you can import any Python module, so the sky is the limit. You also get a simple and intuitive way to work with objects and hierarchical data (like JSON). I do find the language you are proposing very readable and flexible, bringing several advantages to SQL. If you build a parser I would love to bring it to spyql :-) The issues I have brought earlier would not be a problem to spyql since it is a tool to query files and data-streams in the command-line.
In PRQL – A proposal for a better SQL

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.