TOtosh·Jan 27, 2024Submission
quack-reduce: duckdb as a stateless query engine over a data lake
2 points · Open discussionBauplanLabs
quack-reduce
Python2023–2024
BauplanLabs / quack-reduce
quack-reduce
A playground for running duckdb as a stateless query engine over a data lake
2Distinct people
2Mentions
0Explicit recommenders
226GitHub stars
What to know before exploring
What it is
A playground for running duckdb as a stateless query engine over a data lake
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by week2 dated mentions
Observed mentions: 2023-11-17 08:13:33 UTC to 2024-01-27 19:51:33 UTC. Weeks start Monday, UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Nov 13 | 2023-11-13 00:00:00 UTC | 2023-11-20 00:00:00 UTC | 1 | 0 | 1 |
| Nov 20 | 2023-11-20 00:00:00 UTC | 2023-11-27 00:00:00 UTC | 0 | 0 | 0 |
| Nov 27 | 2023-11-27 00:00:00 UTC | 2023-12-04 00:00:00 UTC | 0 | 0 | 0 |
| Dec 4 | 2023-12-04 00:00:00 UTC | 2023-12-11 00:00:00 UTC | 0 | 0 | 0 |
| Dec 11 | 2023-12-11 00:00:00 UTC | 2023-12-18 00:00:00 UTC | 0 | 0 | 0 |
| Dec 18 | 2023-12-18 00:00:00 UTC | 2023-12-25 00:00:00 UTC | 0 | 0 | 0 |
| Dec 25 | 2023-12-25 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 1 | 2024-01-01 00:00:00 UTC | 2024-01-08 00:00:00 UTC | 0 | 0 | 0 |
| Jan 8 | 2024-01-08 00:00:00 UTC | 2024-01-15 00:00:00 UTC | 0 | 0 | 0 |
| Jan 15 | 2024-01-15 00:00:00 UTC | 2024-01-22 00:00:00 UTC | 0 | 0 | 0 |
| Jan 22 | 2024-01-22 00:00:00 UTC | 2024-01-29 00:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
2 sourcesDuckDB use case? I remember being in the same spot. Then I realized "the" use case for DuckDB didn't really exist. There were dozens of use-cases. No wonder I couldn't figure out "the" use case from reading all the articles/blogs/hype people were creating about what they were doing with DuckDB. It seemed like everyone was using it in different ways. It's a data engineering/analytics Swiss army knife.
ETL/ELT, ad-hoc/exploratory, munger/wrangler, edge-analytics, and so on...
> DuckDB will eventually hit a ceiling.
Yeah, probably. Just about every database has hit a ceiling in the past. But then someone comes out with some fantastic new idea to overcome the challenges to some degree. Map-reduce, moving the query to the data instead of the data to the query, serverless, separating storage/compute...
However, what if we start thinking along t
Read the rest
hese lines with DuckDB?
Reading parquet files addresses separating storage and compute. Parquet also provides columnar/row-grouped data giving us push-down predicates (so kinda moving part of the query closer to the data).
We can run a DuckDb instance (EC2/S3) closer to the data so that sorta helps too.
What I'm really excited about using DuckDB in a similar way to map-reduce. What if there was a way to take some SQL's logical plan and turn it into a physical plan that uses compute resources from a pool of (serverless) DuckDB instances. Starting at the leaves of the graph (physical plan) pulling/filtering data from the source (parquet files), and returning their completed work up the branches until it is completed and ready to be served up as results.
I've seen a few examples of this already, but nothing that I would consider production ready. I have a hunch that someone is going to drop such a project on us shortly, and it's going to change a lot of things we have become use to in the data world.
https://github.com/BauplanLabs/quack-reduce
https://www.boilingdata.com/