HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
frankmcsherry
blog
JavaScript2015–2026
frankmcsherry / blog

blog

Some notes on things I find interesting and important.

Productivity
Explore on GitHubRead Hacker News discussions
44Distinct people
79Mentions
2Explicit recommenders
2.1KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Some notes on things I find interesting and important.

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

I recommend Frank McSherry's real-time log of working through AoC 2023 using a SQL dialect:: https://github.com/frankmcsherry/blog/blob/master/posts/2024... The most powerful…

ilya_m used a recommendation phrase. 2 indexed comments match. This is a phrase match, not a verdict.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 12:45:45 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year79 dated mentions

Observed mentions: 2015-08-19 13:05:12 UTC to 2026-06-23 17:32:05 UTC.

0510152015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 14 mentions; 11 submissions, 3 comments20152016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 14 mentions; 12 submissions, 2 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 14 mentions; 7 submissions, 7 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 5 mentions; 2 submissions, 3 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 4 mentions; 1 submissions, 3 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 9 mentions; 4 submissions, 5 comments20202021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 4 mentions; 2 submissions, 2 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 2 mentions; 2 submissions, 0 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 5 mentions; 2 submissions, 3 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 2 mentions; 2 submissions, 0 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC14113
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC14122
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC1477
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC523
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC413
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC945
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC404
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC422
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC202
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC220
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC523
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC220

From the project README

An excerpt from the repository, fetched Nov 13, 2026.

Read the project’s introduction
I am a researcher and computer scientist. I used to work in San Francisco, then I traveled a bit, and I'm now in NYC starting Materialize.io. The following posts are in reverse chronological order: newest posts are first. The content of the oldest posts may no longer be correct (mostly those about pieces of software I've worked on, and their state at some time in the past), but they have been retained for posterity (and some humility). topics date title Timely 2026-03-19 Speeding up timely dataflow by 100x Columnar 2026-03-14 Improving columnar with Claude Columnar, Claude 2026-03-14 Making Field Access Constant-Time in Columnar Columnar 2026-03-01 A Toy Vectorized Interpreter Datalog 2026-01-01 The Demand Transform Datalog 2025-12-23 (Worst-Case Optimal) Relational Programming Datalog 2025-12-14 Evaluating Datatoad Datalog 2025-12-03 Worst-Case Optimal Datalog Datalog, Columnar 2025-11-21 Columnar Worst-Case Optimal Joins Datalog 2025-11-17 Accumulating mid-Join Datalog, Columnar 2025-10-24 Columnar Datatoad Datalog 2025-10-09 Interpreted Datalog Check-in Materialize 2025-09-22 Programming with time in Materialize Datalog 2025-08-24 Checking in on Datatoad Datalog 2025-08-02 Comparing Interpreted Datalogs Materialize 2025-07-21 Materialize R&D Hackathon Datalog 2025-06-03 Datalog in Rust Differential 2025-05-29 Binary joins can be worst-case optimal Timely 2025-05-10 Asynchronous All-Reduce Timely, Differential 2025-05-09 Timely and Differential on WASM Materialize 2025-04-27 Transaction Processing in the Data Plane Bluesky 2025-03-13 Exploring Social Trends on Bluesky McSherry 2024-12-23 A decade in review (2015 - 2024) Materialize 2024-11-25 Understanding Consistency in Materialize E-Graphs 2024-10-19 Understanding E-Graphs Columnar, Timely 2024-10-11 Dataflow and Columns and WASM, Oh My! Columnar 2024-09-10 A Fascinating Diversion into Compression Columnar 2024-08-25 More serial layout in Rust Columnar 2024-08-24 A less unsafe serial layout in Rust Materialize 2024-08-13 Zero-staleness: Like using your primary, but faster Materialize 2024-05-19 Demonstrating Operational Data with SQL Materialize 2024-03-18 Computing and Maintaining Weird (Outer) Joins Materialize 2024-01-02 Materialize and Advent of Code (50/50) Materialize 2023-12-20 Materialize and Memory Materialize 2023-12-19 Doing Business with Recursive SQL Materialize 2023-10-10 Responsiveness and Operational Agility Materialize 2023-09-29 Freshness and Operational Autonomy Materialize 2023-09-19 Consistency and Operational Confidence Materialize 2023-09-17 A Guided Tour of Materialize's Product Principles Materialize 2023-07-21 Capturing Change Data Capture (CDC) Data Materialize 2023-01-04 Advent of Code 2022: In SQL Materialize 2022-12-28 Reasons for Recursion Materialize 2022-12-25 Recursion in Materialize Materialize 2022-07-06 Minesweeping in Materialize Materialize 2021-05-14 Maintaining Joins using Few Resources Differential 2021-04-26 Generalizing Linear Operators Materialize 2021-02-11 Windows enough, and Time Materialize 2020-12-29 Slicing up Temporal Aggregates in Materialize Materialize 2020-11-18 Joins in Materialize Materialize 2020-09-24 Materialize Under the Hood Materialize 2020-08-13 Lateral Joins and Demand-Driven Queries Differential 2020-08-01 Change Data Capture (part 1) Materialize 2020-04-01 Robust Reductions in Materialize Materialize 2020-06-19 Eventual Consistency isn't for Streaming Timely, Differential 2020-06-09 Rust for Data-Intensive Computation Differential 2020-06-06 Sudoku in Differential Dataflow Differential 2020-05-01 Managing Memory in Differential Dataflow Differential 2020-03-26 Upserts in Differential Dataflow Materialize 2020-03-16 Materialize on NYC taxi data Differential 2020-02-15 Differential Dataflow and Calculus Datalog, Differential 2019-09-06 Datalog at SIGMOD and VLDB Timely 2019-08-17 Tracking progress in timely dataflow Materialize 2019-08-03 Debugging dataflows with materialized Differential 2019-06-13 LDBC BI Query 25, developed Differential 2019-05-20 Programming with time Privacy 2019-04-12 Differential privacy and Demographics Differential 2019-02-09 Monoids and Graph Processing Differential 2018-12-30 Tensorflow in Differential Dataflow Differential 2018-11-20 Strings (and beyond) in differential dataflow Timely 2018-09-26 Timely Dataflow Architecture Timely 2018-08-18 Synchronization via Timely Dataflow Datalog, Rust 2018-05-19 A Datalog engine in Rust Differential 2018-05-06 A differential dataflow query processor Privacy 2018-03-11 Things that count Privacy 2018-02-25 Uber's differential privacy .. probably isn't Differential 2018-02-19 World enough, and timely dataflow Differential 2018-02-11 Yahoo's Steaming Benchmark Differential 2017-12-10 Advent of Code Timely 2017-11-08 Timely on Kafka Privacy 2017-10-27 Deep learnings about differential privacy COST, Differential 2017-10-23 COST in the land of differential dataflow Privacy 2017-09-28 The Moral Character of Privacy Technology COST 2017-09-23 COST in the land of databases Differential 2017-09-05 Big Data Stream Managers (BDSM) Differential 2017-08-21 Live PageRanking Timely 2017-07-27 Memory management for big data Differential 2017-07-24 Throughput and latency in differential dataflow Differential 2017-05-06 Specialization in differential dataflow Differential 2017-05-01 Arrangement in differential dataflow Differential 2017-04-24 A TPC-H-like evaluation of differential dataflow Differential 2017-03-28 Differential dataflow 2017 roadmap Differential 2017-03-22 What I'm stuck on Differential 2017-03-01 High-resolution timestamps Differential 2017-02-21 Modular data organization Differential 2017-02-11 Differential dataflow status report Privacy 2017-02-08 Two flavors of differential privacy Privacy 2017-01-26 Dif
Read the full README on GitHub

What Hacker News says

14 sources
geofft·Oct 19, 2017
Yup. See also this recent followup from him: https://github.com/frankmcsherry/blog/blob/master/posts/2017... Literally the papers are "I built a thing, and it gets faster when you do this thing to it." There's no condition on whether the base case is realistic. To be clear, I don't think the authors of the papers he's taking down are bad people, or maliciously trying to get publications for uninteresting work, or too stupid to implement things efficiently on a single core, or anything. I think they're genuinely interested in the work they're doing, but responding to the pressures of the environment that allows them to keep doing that work.
In When the Revolution Came for Amy Cuddy
frankmcsherry·Oct 9, 2017
For further reading about the high quality of database query optimization, and how far back MR et al must have set things, recent SIGMOD work managed to get to within 1000x of a single-threaded implementation (and so, not quite within that of data-parallel systems): https://github.com/frankmcsherry/blog/blob/master/posts/2017... I don't use databases because they are really quite bad at computation. In my opinion, the main recent novelty in query planning has been the work on worst-case optimal joins, stuff like EmptyHeaded[1] and the recent FAQ work[2]. [1]: https://arxiv.org/abs/1503.02368 [2]: https://arxiv.org/abs/1504.04044
In Readings in Database Systems, 5th Edition (2015)
kbenson·Sep 27, 2017
Just to give you an idea of how effective this strategy seems to be, this seems very similar to what the COST in the land of databases submission the other day was talking about, where choosing interesting, fitting algorithms for the problem instead of large distributed but general systems often yields very good results (enough such that a single thread can often beat 100+ core distributed systems). In one of the linked prior blog posts[2] of his he linked to in the beginning of that article, he said this: Using the Hilbert curve layout leads to a very simple compressed representation: having sorted the u64 values it uses to represent edges, one can simply delta-encode them (writing the differences between adjacent values). Because the Hilbert curve teases out locality, the gaps are often quite small, and therefore compress nicely. When I
Read the rest
combined a variable-length integer encoding with gzip, the resulting file was only 45GB; about 2.8 bits per edge). Decompressed (with just the variable-length delta encoding), it is 154GB, which is the representation I used to get the measurements above. He then goes on in his posts to comically trounce some large distributed systems. 1: https://news.ycombinator.com/item?id=15332051 2: https://github.com/frankmcsherry/blog/blob/master/posts/2015...
In Stream VByte: breaking new speed records for integer compression
lambda·Sep 25, 2017
Note that the author himself is working on a framework for distributed differential dataflow computations (dataflow computations in which it is very cheap to make small updates to the inputs and see the differences in the outputs), and he has plans to apply the same metrics (comparing to a single-threaded implementation) to his own work once he gets a little further: https://github.com/frankmcsherry/blog/blob/master/posts/2017... I think that answering the "fastest way to do X in a cluster" question is only particularly interesting if X is something that you need to do in a cluster. But clearly they are testing X that is capable of being done on a mid-range laptop. There's a real question of whether the X that they are testing will actually scale to larger datasets than the laptop (or a decently beefy single node) could handle; it's entir
Read the rest
ely possible that the cluster solution will blow up at that point. And of course, if the single-threaded implementation is faster, then you could presumably just do "X in a cluster" by doing "X on a single node" and using the rest of the nodes to mine your CPU bound cryptocurrency of choice.
In COST in the land of databases
frankmcsherry·Sep 15, 2017
This example isn't about branch prediction (not exactly), but it might still be interesting: https://github.com/frankmcsherry/blog/blob/master/posts/2015... It's about how sorting + random access can be faster than random access, because you introduce locality of reference. And in this case, it absolutely is faster to sort the data first and then do the work, even counting the sorting. Edit: Aw crap, adrianN beat me to this a few screenfuls down, sorry! But, it is a different post, so maybe this is still helpful. :D
In Why is it faster to process a sorted array than an unsorted array? (2012)
frankmcsherry·Apr 29, 2017
No. At least, not in the case of graph computation. This was the conventional wisdom with some people, but circa a year or so ago when we did some measurements, no. https://github.com/frankmcsherry/blog/blob/master/posts/2015... https://github.com/frankmcsherry/blog/blob/master/posts/2015... If you don't want to read everything (but you should, before trusting the conclusions), the relevant measurements are here: https://github.com/frankmcsherry/blog/blob/master/posts/2015... Edit: Reading the paper, they cite these measurements and present their work as "only [..] 8.8× slower than a distributed in-memory" system (GraM, a different system than timely dataflow). The quote is on the first page.
In A Trillion Edge Graph on a Single Commodity Node

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.