ArroyoSystems

arroyo

Rust2023–2024

ArroyoSystems / arroyo

arroyo

Distributed stream processing engine in Rust

Data & Databases Systems & Infrastructure

Explore on GitHub Read Hacker News discussions

5Distinct people

11Mentions

0Explicit recommenders

5.1KGitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Distributed stream processing engine in Rust

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month11 dated mentions

Observed mentions: 2023-04-05 17:32:13 UTC to 2024-03-19 13:01:30 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
Apr 2023	2023-04-01 00:00:00 UTC	2023-05-01 00:00:00 UTC	2	2	0
May 2023	2023-05-01 00:00:00 UTC	2023-06-01 00:00:00 UTC	0	0	0
Jun 2023	2023-06-01 00:00:00 UTC	2023-07-01 00:00:00 UTC	1	1	0
Jul 2023	2023-07-01 00:00:00 UTC	2023-08-01 00:00:00 UTC	0	0	0
Aug 2023	2023-08-01 00:00:00 UTC	2023-09-01 00:00:00 UTC	2	0	2
Sep 2023	2023-09-01 00:00:00 UTC	2023-10-01 00:00:00 UTC	0	0	0
Oct 2023	2023-10-01 00:00:00 UTC	2023-11-01 00:00:00 UTC	1	0	1
Nov 2023	2023-11-01 00:00:00 UTC	2023-12-01 00:00:00 UTC	0	0	0
Dec 2023	2023-12-01 00:00:00 UTC	2024-01-01 00:00:00 UTC	0	0	0
Jan 2024	2024-01-01 00:00:00 UTC	2024-02-01 00:00:00 UTC	3	0	3
Feb 2024	2024-02-01 00:00:00 UTC	2024-03-01 00:00:00 UTC	0	0	0
Mar 2024	2024-03-01 00:00:00 UTC	2024-04-01 00:00:00 UTC	2	0	2

What Hacker News says

11 sources

gbto·Mar 19, 2024

Currently working on an integration with NATS jetstream, see https://github.com/ArroyoSystems/arroyo/compare/master...gbt... I'll be refactoring it following arroyo 0.10 release, and add some tests. Happy to get your feedbacks (or even better your help) if something is missing

In Building a streaming SQL engine with Arrow and DataFusion

esafak·Mar 18, 2024

I don't see any PRs: https://github.com/ArroyoSystems/arroyo/pulls There is a request: https://github.com/ArroyoSystems/arroyo/issues/162 In any case, I look forward to it.

In Building a streaming SQL engine with Arrow and DataFusion

necubi·Jan 30, 2024

We have one very slow job (our Rust CI, for example: https://github.com/ArroyoSystems/arroyo/actions/runs/7702793...) and a bunch of little jobs that take a few seconds (checking lints, etc.). We never bothered to switch those over because they complete quickly on github and fit within our included runner minutes. I should say we also use BuiltJet for docker builds because of their arm support. Now that ubicloud has arm we may switch those over as well.

In Show HN: Open-source x64 and Arm GitHub runners

necubi·Jan 30, 2024

We've been using Ubicloud builders for our Rust project [0] for several months, and it's worked very well. We've seen CI times go from 10-15 minutes to 6-7, and our bill has gone from $300/month to $30. One counter-intuitive thing we found is that it's slow to save and restore caches, but the machines have good CPU, so for us it's been faster to disable cache entirely and just redo everything on each build. [0] https://github.com/ArroyoSystems/arroyo

In Show HN: Open-source x64 and Arm GitHub runners

necubi·Jan 20, 2024

Yep. There are always going to be constraints about how well a system like clickhouse can support arbitrary joins. Queries in clickhouse are fast because the data is laid out in such a way that it can minimize how much it needs to read. Part of this is the columnar layout that means it can avoid reading columns that are not involved in the query. However it’s also able to push query predicates into the table scan, using metadata (like bloom filters) that tell it what values are in each chunk of data. But for joins, you typically end up needing to read all of the data and materialize it in memory. For realtime joins the best option is to do it in a steaming fashion on ingestion, for example in a system like Flink or Arroyo [0], which I work on. [0] https://github.com/ArroyoSystems/arroyo

In Let's talk about joins

necubi·Oct 21, 2023

I'm with Arroyo [0] — thanks for the mention! I'd be interested to see someone from Epsio chime in with where exactly they're positioning, but you're right that this is (recently) a very crowded space. I think you can somewhat arbitrarily draw a line between systems like Materialize/RisingWave that are focused on materialized view maintenance (often reading change feeds from your OLTP database) and stream processors like Flink/Arroyo that are focused on supporting more operational use cases and work with a lager variety of sources and sinks. Epsio seems like it's working primarily in the former mold, with fast incremental computation of materialized views. Unlike Materialize/RisingWave it seems to be designed to run in front of your database, with all queries going through it. ReadySet is a really cool project in the query caching/materia

Read the rest

lized view space that I think doesn't get enough attention. Rather than making you define your materialized views ahead of time, it acts as an automated caching layer on top of postgres/mysql that performs incremental computation for components of query graphs. As someone who's been in the streaming space for years, it's really exciting to see so much energy in the space in the past couple of years after a long period of stagnation, with everyone trying to figure out the programming and deployment models that make the most sense. Most folks right now are gravitating towards materialized views as the model, largely because it's easy and familar for users. But ultimately I think this approach will end up too limited for most use cases and will remain valuable but somewhat niche. [0] https://github.com/ArroyoSystems/arroyo

In We Built a Streaming SQL Engine

scott_s·Aug 1, 2023

Interesting - I looked into your code a bit. I found your window aggregation library [1]. You may be interested in looking into the Rust implementation of some of the research work I've been a part of [2]. In Flink, I believe the reason they need to implement their own backpressure system is that they multiplex TCP connections. That is, they have multiple logical streams flowing through a single TCP connection. If that's the case, you need to do some work to 1) detect which logical stream is the one that's blocking, and 2) don't block because other logical streams may be able to use the active TCP connection. Thinking it through, I think what Flink's approach buys is not necessarily better performance, but better just a manageable number of connections. That is, imagine you have a process P1 with operators A, B and C. And then P2 has D, E

Read the rest

, F. Now imagine that this is a shuffle, where A, B and C are fully connected to D, E and F. In my old system, you would have 9 TCP connections. In Flink, you will have 1. [1] https://github.com/ArroyoSystems/arroyo/blob/master/arroyo-w... [2] https://github.com/IBM/sliding-window-aggregators/tree/maste...

In Query Engines: Push vs. Pull

necubi·Aug 1, 2023

This is an interesting point. In Arroyo [0] (the streaming engine I work on) we also use a simple TCP-based backpressure mechanism: we have NxM TCP connections between each operator subtask, behind small in-memory queues. This essentially is relying on the kernel's flow control and backpressure mechanisms instead of building a custom network stack like Flink does. This has worked well in practice, but I think does leave some performance on the table and likely incurrs higher resource utilization. Flink for example uses a system of "floating" buffers that can be moved between streams as needed to minimize the total number of buffers needed. However it may be that Linux's network stack has gotten good enough at this point that it makes sense to just rely on it directly. I wish there was more research out there on how these sorts of network

Read the rest

stacks should be built on modern hardware and OSes. (semi-relatedly, was checking out your CV; a bunch of great looking papers there that I'll definitely be checking out!) [0] https://github.com/ArroyoSystems/arroyo

In Query Engines: Push vs. Pull

necubi·Jun 6, 2023Submission

Show HN: Arroyo – Write SQL on streaming data

Hey HN, Arroyo is a modern, open-source stream processing engine, that lets anyone write complex queries on event streams just by writing SQL—windowing, aggregating, and joining events with sub-second latency. Today data processing typically happens in batch data warehouses like BigQuery and Snowflake despite the fact that most of the data is coming in as streams. Data teams have to build complex orchestration systems to handle late-arriving data and job failures while trying to minimize latency. Stream processing offers an alternative approach, where the query is compiled into a streaming program that constantly updates as new data comes in, providing low-latency results as soon as the data is available. I started the Arroyo project after spending the past five years building real-time platforms at Lyft and Splunk. I saw first hand how

Read the rest

hard it is for users to build correct, reliable pipelines on top of existing systems like Flink and Spark Streaming, and how hard those pipelines are to operate for infra teams. I saw the need for a new system that would be easy enough for any data team to adopt, built on modern foundations and with the lessons of the past decade of research and industry development. Arroyo works by taking SQL queries and compiling them into an optimized streaming dataflow program, a distributed DAG of computation with nodes that read from sources (like Kafka), perform stateful computations, and eventually write results to sinks. That state is consistently snapshotted using a variation of the Chandy-Lamport checkpointing algorithm for fault-tolerance and to enable fast rescaling and updates of the pipelines. The entire system is easy to self-host on Kubernetes and Nomad. See it in action here: https://www.youtube.com/watch?v=X1Nv0gQy9TA or follow the getting started guide (https://doc.arroyo.dev/getting-started) to run it locally.

115 points · Open discussion

tim_sw·Apr 13, 2023Submission

Distributed stream processing engine written in Rust

1 points · Open discussion

tim_sw·Apr 5, 2023Submission

Arroyo – Distributed stream processing engine written in Rust

6 points · Open discussion