RErefulgentis·Dec 16, 2023
I grep'd Soundcloud, then clicked the <a href> wrapping it, it linked here: https://github.com/soundcloud/roshi
(tl;Dr: time-series event storage via a LWW-element-set)
In Causal Treessoundcloud
roshi
Go2014–2023
soundcloud / roshi
roshi
Roshi is a large-scale CRDT set implementation for timestamped events.
10Distinct people
11Mentions
0Explicit recommenders
3.2KGitHub stars
What to know before exploring
What it is
Roshi is a large-scale CRDT set implementation for timestamped events.
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year11 dated mentions
Observed mentions: 2014-05-14 08:37:44 UTC to 2023-12-16 20:22:36 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2014 | 2014-01-01 00:00:00 UTC | 2015-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2015 | 2015-01-01 00:00:00 UTC | 2016-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 2 | 1 | 1 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
11 sourcesSoundCloud's Roshi database is built on LWW-Element-Sets, but doesn't have an explicitly CRDT-based API. It looks like they no longer use it internally, however.
https://developers.soundcloud.com/blog/roshi-a-crdt-system-f...
https://github.com/soundcloud/roshi
In Ask HN: Are there any databases backed by CRDTs?Roshi: A large-scale CRDT set implementation for timestamped events
73 points · Open discussionThis approach was used successfully in bespoke data services like Roshi https://github.com/soundcloud/roshi, the design seems sound, nice to hear.
In KeyDB: A Multithreaded Redis ForkSoundcloud/roshi: large-scale CRDT set implementation for timestamped events
4 points · Open discussionI am not that into this topic, but could you compare DL to Soundclouds Roshi[0], and if not, whats different? Thanks!
[0] https://github.com/soundcloud/roshi
In Twitter open-sources a high-performance replicated log service>I still fail to see how you need 200 people to run a CRUD app that streams audio.
Have you used SoundCloud? SoundCloud is not a CRUD app. SoundCloud is a full blown social network. Just take a look at their GitHub page: https://github.com/soundcloud?utf8=%E2%9C%93&query=+only%3As...
I'm reminded of Roshi, their distributed/crdt Redis solution, they open sourced (https://github.com/soundcloud/roshi) a while back just to deal with problems of their scale. CRDTs and Distrubuted programming is not something they teach at coding bootcamps. You should really take a look at some of their developer talks and get a sense of their scaling pains before you assume that SoundCloud is built by 200 fresh-out-of bootcampers (Also, those 200 aren't exclusively engineers, you need a sizable marketing/community team to manage all that global talent on your
Read the rest
platform).
Even a feed, where one user can publish to millions others, is non-trivial at scale. I think you are seriously underestimating the amount of work that goes into building something like SoundCloud.
Ignoring this - it doesn't seem SC has ever been in the green so comparing to relatively simple models of DeviantArt or Craigslist doesn't make sense. DeviantArt doesn't have the target of the MPAA on its back. SoundCloud will never be as cheap as Craigslist to run.
Also, SoundCloud hasn't been acquired - I'm not sure where you are getting this info. The same guys who ran the ship in its "humble beginnings" are running it now.
Roshi: large-scale CRDT set implementation for timestamped events
26 points · Open discussionMaybe check this out?
https://github.com/soundcloud/roshi
Roshi is basically a high-performance index for timestamped data. It's designed to sit in the critical (request) path of your application or service. The originating use case is the SoundCloud stream; see this blog post for details.
In Ask HN: What DB to use for huge time series?> Do you vendor [transitive dependencies] too?
Yes. Here's an example: https://github.com/soundcloud/roshi
In Show HN: Duo – a next-generation package manager for the front endI'm trying to understand CRDTs, and Roshi provides an implementation of how CRDTs can be put to use in real-world applications. I've assess the documentation on Roshi and would like assent or clarification if I'm wrong.
After glossing over the readme of github.com/soundcloud/roshi I came away with the impression that Roshi's LWW-element-set implementation does not strictly adhere to the qualities of a CRDT, mainly that operations must be commutative.
Roshi documents two uses-cases:
If first we apply an add operation to the set, then apply a remove operation with the same tuple, the resulting set will ignore the remove operation:
A(a,1) R() + remove(a,1) = A(a,1) R()
If we apply the remove operation, then next the add operation, the resulting set will ignore the add operation:
A() R(a,1) + add(a,1) = A() R(a,1)
This means the resulting
Read the rest
state of the set depends on the order of operations, which violates the commutative property of CRDTs.
As a consequence, if we assume this CRDT is replicated on each redis node in a Roshi cluster, then there is a case (admittedly rare and short-lived) where a Roshi cluster cannot be eventually consistent:
Given no more future operations on a set with key K, if,
node 1 contains a key K with set A(a,1) R(), and
node 2 contains a key K with set A() R(a,1), then the system cannot be eventually consistent.
Roshi seems to have implemented a weak form of CRDT, which under rare and short-lived situations impedes eventual consistent. But given the operational realities, this is totally acceptable.