Transaction Processing in the Data Plane
3 points · Open discussionfrankmcsherry
blog
JavaScript2015–2026
frankmcsherry / blog
blog
Some notes on things I find interesting and important.
44Distinct people
79Mentions
2Explicit recommenders
2.1KGitHub stars
What to know before exploring
What it is
Some notes on things I find interesting and important.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
I recommend Frank McSherry's real-time log of working through AoC 2023 using a SQL dialect:: https://github.com/frankmcsherry/blog/blob/master/posts/2024... The most powerful…
ilya_m used a recommendation phrase. 2 indexed comments match. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- rust-lang/datafrog · linked in 2 of the same discussions
- timelydataflow/differential-dataflow · linked in 2 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year79 dated mentions
Observed mentions: 2015-08-19 13:05:12 UTC to 2026-06-23 17:32:05 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2015 | 2015-01-01 00:00:00 UTC | 2016-01-01 00:00:00 UTC | 14 | 11 | 3 |
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 14 | 12 | 2 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 14 | 7 | 7 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 5 | 2 | 3 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 4 | 1 | 3 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 9 | 4 | 5 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 4 | 0 | 4 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 4 | 2 | 2 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 2 | 2 | 0 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 5 | 2 | 3 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 2 | 2 | 0 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
I am a researcher and computer scientist. I used to work in San Francisco, then I traveled a bit, and I'm now in NYC starting Materialize.io.
The following posts are in reverse chronological order: newest posts are first. The content of the oldest posts may no longer be correct (mostly those about pieces of software I've worked on, and their state at some time in the past), but they have been retained for posterity (and some humility).
topics
date
title
Timely
2026-03-19
Speeding up timely dataflow by 100x
Columnar
2026-03-14
Improving columnar with Claude
Columnar, Claude
2026-03-14
Making Field Access Constant-Time in Columnar
Columnar
2026-03-01
A Toy Vectorized Interpreter
Datalog
2026-01-01
The Demand Transform
Datalog
2025-12-23
(Worst-Case Optimal) Relational Programming
Datalog
2025-12-14
Evaluating Datatoad
Datalog
2025-12-03
Worst-Case Optimal Datalog
Datalog, Columnar
2025-11-21
Columnar Worst-Case Optimal Joins
Datalog
2025-11-17
Accumulating mid-Join
Datalog, Columnar
2025-10-24
Columnar Datatoad
Datalog
2025-10-09
Interpreted Datalog Check-in
Materialize
2025-09-22
Programming with time in Materialize
Datalog
2025-08-24
Checking in on Datatoad
Datalog
2025-08-02
Comparing Interpreted Datalogs
Materialize
2025-07-21
Materialize R&D Hackathon
Datalog
2025-06-03
Datalog in Rust
Differential
2025-05-29
Binary joins can be worst-case optimal
Timely
2025-05-10
Asynchronous All-Reduce
Timely, Differential
2025-05-09
Timely and Differential on WASM
Materialize
2025-04-27
Transaction Processing in the Data Plane
Bluesky
2025-03-13
Exploring Social Trends on Bluesky
McSherry
2024-12-23
A decade in review (2015 - 2024)
Materialize
2024-11-25
Understanding Consistency in Materialize
E-Graphs
2024-10-19
Understanding E-Graphs
Columnar, Timely
2024-10-11
Dataflow and Columns and WASM, Oh My!
Columnar
2024-09-10
A Fascinating Diversion into Compression
Columnar
2024-08-25
More serial layout in Rust
Columnar
2024-08-24
A less unsafe serial layout in Rust
Materialize
2024-08-13
Zero-staleness: Like using your primary, but faster
Materialize
2024-05-19
Demonstrating Operational Data with SQL
Materialize
2024-03-18
Computing and Maintaining Weird (Outer) Joins
Materialize
2024-01-02
Materialize and Advent of Code (50/50)
Materialize
2023-12-20
Materialize and Memory
Materialize
2023-12-19
Doing Business with Recursive SQL
Materialize
2023-10-10
Responsiveness and Operational Agility
Materialize
2023-09-29
Freshness and Operational Autonomy
Materialize
2023-09-19
Consistency and Operational Confidence
Materialize
2023-09-17
A Guided Tour of Materialize's Product Principles
Materialize
2023-07-21
Capturing Change Data Capture (CDC) Data
Materialize
2023-01-04
Advent of Code 2022: In SQL
Materialize
2022-12-28
Reasons for Recursion
Materialize
2022-12-25
Recursion in Materialize
Materialize
2022-07-06
Minesweeping in Materialize
Materialize
2021-05-14
Maintaining Joins using Few Resources
Differential
2021-04-26
Generalizing Linear Operators
Materialize
2021-02-11
Windows enough, and Time
Materialize
2020-12-29
Slicing up Temporal Aggregates in Materialize
Materialize
2020-11-18
Joins in Materialize
Materialize
2020-09-24
Materialize Under the Hood
Materialize
2020-08-13
Lateral Joins and Demand-Driven Queries
Differential
2020-08-01
Change Data Capture (part 1)
Materialize
2020-04-01
Robust Reductions in Materialize
Materialize
2020-06-19
Eventual Consistency isn't for Streaming
Timely, Differential
2020-06-09
Rust for Data-Intensive Computation
Differential
2020-06-06
Sudoku in Differential Dataflow
Differential
2020-05-01
Managing Memory in Differential Dataflow
Differential
2020-03-26
Upserts in Differential Dataflow
Materialize
2020-03-16
Materialize on NYC taxi data
Differential
2020-02-15
Differential Dataflow and Calculus
Datalog, Differential
2019-09-06
Datalog at SIGMOD and VLDB
Timely
2019-08-17
Tracking progress in timely dataflow
Materialize
2019-08-03
Debugging dataflows with materialized
Differential
2019-06-13
LDBC BI Query 25, developed
Differential
2019-05-20
Programming with time
Privacy
2019-04-12
Differential privacy and Demographics
Differential
2019-02-09
Monoids and Graph Processing
Differential
2018-12-30
Tensorflow in Differential Dataflow
Differential
2018-11-20
Strings (and beyond) in differential dataflow
Timely
2018-09-26
Timely Dataflow Architecture
Timely
2018-08-18
Synchronization via Timely Dataflow
Datalog, Rust
2018-05-19
A Datalog engine in Rust
Differential
2018-05-06
A differential dataflow query processor
Privacy
2018-03-11
Things that count
Privacy
2018-02-25
Uber's differential privacy .. probably isn't
Differential
2018-02-19
World enough, and timely dataflow
Differential
2018-02-11
Yahoo's Steaming Benchmark
Differential
2017-12-10
Advent of Code
Timely
2017-11-08
Timely on Kafka
Privacy
2017-10-27
Deep learnings about differential privacy
COST, Differential
2017-10-23
COST in the land of differential dataflow
Privacy
2017-09-28
The Moral Character of Privacy Technology
COST
2017-09-23
COST in the land of databases
Differential
2017-09-05
Big Data Stream Managers (BDSM)
Differential
2017-08-21
Live PageRanking
Timely
2017-07-27
Memory management for big data
Differential
2017-07-24
Throughput and latency in differential dataflow
Differential
2017-05-06
Specialization in differential dataflow
Differential
2017-05-01
Arrangement in differential dataflow
Differential
2017-04-24
A TPC-H-like evaluation of differential dataflow
Differential
2017-03-28
Differential dataflow 2017 roadmap
Differential
2017-03-22
What I'm stuck on
Differential
2017-03-01
High-resolution timestamps
Differential
2017-02-21
Modular data organization
Differential
2017-02-11
Differential dataflow status report
Privacy
2017-02-08
Two flavors of differential privacy
Privacy
2017-01-26
Dif
Read the full README on GitHubWhat Hacker News says
79 sourcesProgress in Graph Processing (2015)
3 points · Open discussion> which is more efficient than "hacking it" with recursive queries in a relational db
It seems to me that the way recursive CTEs were originally defined is the biggest reason that relational databases haven't been more successful with users who need to run serious graph workloads - in Frank McSherry's words:
> As it turns out, WTIH RECURSIVE has a bevy of limitations and mysterious semantics (four pages of limitations in the version of the standard I have, and I still haven't found the semantics yet). I certainly cannot enumerate, or even understand the full list [...] There are so many things I don't understand here.
https://github.com/frankmcsherry/blog/blob/master/posts/2022...
In Ask HN: What's your experience with using graph databases for agentic use-cases?related: McSherry's preceding blog post was all about demonstrating how binary joins can achieve worst-case optimal runtime, given suitable adjustments to the query plan.
- https://github.com/frankmcsherry/blog/blob/master/posts/2025...
In Datalog in RustI recommend Frank McSherry's real-time log of working through AoC 2023 using a SQL dialect:: https://github.com/frankmcsherry/blog/blob/master/posts/2024...
The most powerful primitive that the dialect has is (mutual) recursion, which is effectively a mechanism for outputting a fixed-point of a query.
In Advent of Code 2024 in pure SQLAdvent of Code 2023 in Recursive SQL
2 points · Open discussion> Datalog has great compositionality, which is one of SQL's big weaknesses
There's a good write-up on this aspect (amongst others) here: https://www.scattered-thoughts.net/writing/against-sql/
> recursive queries are completely natural in Datalog, whereas in SQL they are bolted on with recursive common table expressions
And similarly: https://github.com/frankmcsherry/blog/blob/master/posts/2022...
In Introduction to DatalogThis reminds me of a great blog post by Frank McSherry (Materialize, timely dataflow, etc) talking about how using the right tools on a laptop could beat out a bunch of these JVM distributed querying tools because... data locality basically.
https://github.com/frankmcsherry/blog/blob/master/posts/2015...
In Big data is deadRust for Data-Intensive Computation (2020)
1 points · Open discussionCost in the Land of Databases (2017)
1 points · Open discussionFunny you say that, the Rust borrow checker tried to adopt differential-dataflow directly - the library behind this language - but Frank McSherry told them it's too complicated, let me build you an easier-to-use Datalog instead [blog]. So he built [datafrog] which is at the heart of the borrow checker today.
I'm not sure what the delta between Salsa and differential-dataflow is. My experience with differential-dataflow is that it's also galaxy brain.
[blog]: https://github.com/frankmcsherry/blog/blob/master/posts/2018...
[datafrog]: https://github.com/rust-lang/datafrog
In Differential Datalog: a programming language for incremental computationOh, cool!
And yeah, licenses can be challenging and frustrating, especially the first time you release a major project.
I am really super excited by the idea of embedded Datalog in Rust. I sometimes run into situations where I need something that fits in that awkward gap between SQL and Prolog. I want more expressiveness, better composability, and better graph support than SQL. But I also want finite-sized results that I can materialize in bounded time.
There has been some very neat work with incrementally-updated Datalog in the Rust community. For example, I think Datafrog is really neat: https://github.com/frankmcsherry/blog/blob/master/posts/2018... But it's great to see more cool projects in this space, so thank you.
In Show HN: Cozo – new Graph DB with Datalog, embedded like SQLiteShowing the 15 most recent of 79 sources. Enable JavaScript to search and browse every indexed mention.