TAtanelpoder·Nov 10, 2025Submission
Turning PySpark into a Universal DataFrame API
3 points · Open discussioneakmanrq
sqlframe
Python2024–2025
eakmanrq / sqlframe
sqlframe
Turning PySpark Into a Universal DataFrame API
3Distinct people
3Mentions
1Explicit recommenders
544GitHub stars
What to know before exploring
What it is
Turning PySpark Into a Universal DataFrame API
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
hey, thanks for the shoutout! I love the idea, effectively allowing going towards a direction where the right platform for the right job is used, and it is very much in line…
karakanb used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by month3 dated mentions
Observed mentions: 2024-05-21 01:09:29 UTC to 2025-11-10 18:02:37 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| May 2024 | 2024-05-01 00:00:00 UTC | 2024-06-01 00:00:00 UTC | 1 | 1 | 0 |
| Jun 2024 | 2024-06-01 00:00:00 UTC | 2024-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2024 | 2024-07-01 00:00:00 UTC | 2024-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2024 | 2024-08-01 00:00:00 UTC | 2024-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2024 | 2024-09-01 00:00:00 UTC | 2024-10-01 00:00:00 UTC | 0 | 0 | 0 |
| Oct 2024 | 2024-10-01 00:00:00 UTC | 2024-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2024 | 2024-11-01 00:00:00 UTC | 2024-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2024 | 2024-12-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 1 | 0 | 1 |
| Jan 2025 | 2025-01-01 00:00:00 UTC | 2025-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2025 | 2025-02-01 00:00:00 UTC | 2025-03-01 00:00:00 UTC | 0 | 0 | 0 |
| Mar 2025 | 2025-03-01 00:00:00 UTC | 2025-04-01 00:00:00 UTC | 0 | 0 | 0 |
| Apr 2025 | 2025-04-01 00:00:00 UTC | 2025-05-01 00:00:00 UTC | 0 | 0 | 0 |
| May 2025 | 2025-05-01 00:00:00 UTC | 2025-06-01 00:00:00 UTC | 0 | 0 | 0 |
| Jun 2025 | 2025-06-01 00:00:00 UTC | 2025-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2025 | 2025-07-01 00:00:00 UTC | 2025-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2025 | 2025-08-01 00:00:00 UTC | 2025-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2025 | 2025-09-01 00:00:00 UTC | 2025-10-01 00:00:00 UTC | 0 | 0 | 0 |
| Oct 2025 | 2025-10-01 00:00:00 UTC | 2025-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2025 | 2025-11-01 00:00:00 UTC | 2025-12-01 00:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
3 sourceshey, thanks for the shoutout!
I love the idea, effectively allowing going towards a direction where the right platform for the right job is used, and it is very much in line with where we are taking things towards. Another interesting project in that spirit is sqlframe: https://github.com/eakmanrq/sqlframe
In Show HN: I built an open-source data pipeline tool in GoShow HN: SQLFrame – I ran PySpark without Spark on a SQL database
Recently I open-sourced SQLFrame, a DataFrame library that implements the PySpark DataFrame API but removes Spark as a dependency. It does this by generating the corresponding SQL for the DataFrame operations using SQLGlot. Since the output is SQL this also means that the PySpark DataFrame API can now be used directly against other databases without the Spark middleman.
I built this because of two common problems I have faced in my career:
1. I prefer to write complex pipelines in PySpark but they can be hard to read for SQL-proficient co-workers. Therefore I find myself in a tradeoff between maintainability and accessibility.
2. I really enjoy using the PySpark DataFrame API but not every project requires Spark and therefore I'm not able to use the DataFrame library I am most proficient in.
The library currently focuses on transformation
Read the rest
pipelines (reading from and writing to tables) and data analysis as key use cases. It does offer some ability to read from files directly but they must be small although this can be improved over time if there is demand for it.
SQLFrame currently supports BigQuery, DuckDB, and Postgres with Clickhouse, Redshift, Snowflake, Spark, and Trino in development or planned. You can use the "Standalone" session to test running against any engine supported by SQLGlot but there could be issues with more advanced functions that will be resolved once officially supported by SQLFrame.
Blog post for more details: https://github.com/eakmanrq/sqlframe/blob/main/blogs/sqlfram...
Would love to answer any questions or hear any feedback you may have!