HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
ArroyoSystems
arroyo
Rust2023–2024
ArroyoSystems / arroyo

arroyo

Distributed stream processing engine in Rust

5Distinct people
11Mentions
0Explicit recommenders
5.1KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Distributed stream processing engine in Rust

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 4:43:50 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month11 dated mentions

Observed mentions: 2023-04-05 17:32:13 UTC to 2024-03-19 13:01:30 UTC.

01232023-04-01 00:00:00 UTC to 2023-05-01 00:00:00 UTC (exclusive): 2 mentions; 2 submissions, 0 commentsApr 20232023-05-01 00:00:00 UTC to 2023-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-06-01 00:00:00 UTC to 2023-07-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2023-07-01 00:00:00 UTC to 2023-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-08-01 00:00:00 UTC to 2023-09-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2023-09-01 00:00:00 UTC to 2023-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsSep 20232023-10-01 00:00:00 UTC to 2023-11-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-11-01 00:00:00 UTC to 2023-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-12-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2024-02-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2024-02-01 00:00:00 UTC to 2024-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-03-01 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 commentsMar 2024

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Apr 20232023-04-01 00:00:00 UTC2023-05-01 00:00:00 UTC220
May 20232023-05-01 00:00:00 UTC2023-06-01 00:00:00 UTC000
Jun 20232023-06-01 00:00:00 UTC2023-07-01 00:00:00 UTC110
Jul 20232023-07-01 00:00:00 UTC2023-08-01 00:00:00 UTC000
Aug 20232023-08-01 00:00:00 UTC2023-09-01 00:00:00 UTC202
Sep 20232023-09-01 00:00:00 UTC2023-10-01 00:00:00 UTC000
Oct 20232023-10-01 00:00:00 UTC2023-11-01 00:00:00 UTC101
Nov 20232023-11-01 00:00:00 UTC2023-12-01 00:00:00 UTC000
Dec 20232023-12-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
Jan 20242024-01-01 00:00:00 UTC2024-02-01 00:00:00 UTC303
Feb 20242024-02-01 00:00:00 UTC2024-03-01 00:00:00 UTC000
Mar 20242024-03-01 00:00:00 UTC2024-04-01 00:00:00 UTC202

What Hacker News says

2 sources
Interesting - I looked into your code a bit. I found your window aggregation library [1]. You may be interested in looking into the Rust implementation of some of the research work I've been a part of [2]. In Flink, I believe the reason they need to implement their own backpressure system is that they multiplex TCP connections. That is, they have multiple logical streams flowing through a single TCP connection. If that's the case, you need to do some work to 1) detect which logical stream is the one that's blocking, and 2) don't block because other logical streams may be able to use the active TCP connection. Thinking it through, I think what Flink's approach buys is not necessarily better performance, but better just a manageable number of connections. That is, imagine you have a process P1 with operators A, B and C. And then P2 has D, E
Read the rest
, F. Now imagine that this is a shuffle, where A, B and C are fully connected to D, E and F. In my old system, you would have 9 TCP connections. In Flink, you will have 1. [1] https://github.com/ArroyoSystems/arroyo/blob/master/arroyo-w... [2] https://github.com/IBM/sliding-window-aggregators/tree/maste...
In Query Engines: Push vs. Pull
This is an interesting point. In Arroyo [0] (the streaming engine I work on) we also use a simple TCP-based backpressure mechanism: we have NxM TCP connections between each operator subtask, behind small in-memory queues. This essentially is relying on the kernel's flow control and backpressure mechanisms instead of building a custom network stack like Flink does. This has worked well in practice, but I think does leave some performance on the table and likely incurrs higher resource utilization. Flink for example uses a system of "floating" buffers that can be moved between streams as needed to minimize the total number of buffers needed. However it may be that Linux's network stack has gotten good enough at this point that it makes sense to just rely on it directly. I wish there was more research out there on how these sorts of network
Read the rest
stacks should be built on modern hardware and OSes. (semi-relatedly, was checking out your CV; a bunch of great looking papers there that I'll definitely be checking out!) [0] https://github.com/ArroyoSystems/arroyo
In Query Engines: Push vs. Pull

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.