omnistreams

omnistreams-spec

Learning & Reference2019–2022

omnistreams / omnistreams-spec

omnistreams-spec

WIP Specification for "universal" (language- and transport-agnostic) data streams

Learning & Reference

Explore on GitHub Read Hacker News discussions

2Distinct people

8Mentions

0Explicit recommenders

10GitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

WIP Specification for "universal" (language- and transport-agnostic) data streams

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

reactive-streams/reactive-streams-jvm · linked in 3 of the same discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year8 dated mentions

Observed mentions: 2019-06-01 19:30:06 UTC to 2022-02-12 19:07:42 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Comments
2019	2019-01-01 00:00:00 UTC	2020-01-01 00:00:00 UTC	2	2
2020	2020-01-01 00:00:00 UTC	2021-01-01 00:00:00 UTC	4	4
2021	2021-01-01 00:00:00 UTC	2022-01-01 00:00:00 UTC	0	0
2022	2022-01-01 00:00:00 UTC	2023-01-01 00:00:00 UTC	2	2

What Hacker News says

8 sources

anderspitman·Feb 12, 2022

My personal browser streaming TL;DR goes something like this: * Start with SSE * If you need to send binary data, use long polling or WebSockets * If you need fast bidi streaming, use WebSockets * If you need backpressure and multiplexing for WebSockets, use RSocket or omnistreams[1] (one of my projects). * Make sure you account for SSE browser connection limits, preferably by minimizing the number of streams needed, or by using HTTP/2 (mind head-of-line blocking) or splitting your HTTP/1.1 backend across multiple domains and doing round-robin on the frontend. [0]: https://rsocket.io/ [1]: https://github.com/omnistreams/omnistreams-spec

In Server-Sent Events: an alternative to WebSockets

anderspitman·Jan 11, 2022

I built omnistreams[0] primarily because of the lack of backpressure in browser WebSockets (lots of background information and references in that README). It's what fibridge[1] is built on. We've been using it in production for over 2 years, but I never ended up trying to push omnistreams as a thing. I believe the Rust implementation is actually behind the spec a bit, and the spec itself probably needs some work. At the end of the day I think RSocket is probably the way to go for most people, though the simplicity of omnistreams is still appealing to me. EDIT: I just learned about WebSocketStreams[2] from another comment[3] and sounds like they may solve the backpressure issue natively. [0]: https://github.com/omnistreams/omnistreams-spec [1]: https://iobio.io/2019/06/12/introducing-fibridge/ [2]: https://web.dev/websocketstream/ [3]: htt

Read the rest

ps://news.ycombinator.com/item?id=29894938

In The WebSocket Handbook

anderspitman·Sep 2, 2020

I've been working on solutions in this space for a couple years now. IMO making data on your local device available to others via HTTP range requests is the sweet spot between storing your data in the cloud and going full p2p. Here's a couple of my projects: https://patchbay.pub/ code[0] Sender: curl https://patchbay.pub/random-channel/filename.bin --data-binary @filename.bin Receiver: curl -O https://patchbay.pub/random-channel/filename.bin https://fbrg.xyz code[1] This one works in the browser. You select a file and it gives you a link you can share with others. The underlying tech[2] is more complicated (relies on WebSockets) than patchbay, but the server is currently more reliable. I'm in the process of improving the patchbay server to be more robust, and I think it will eventually replace fibridge completely. My current projec

Read the rest

t is building something along the lines of Google Drive, but much simpler and designed to be self-hosted from within your home. [0]: https://github.com/patchbay-pub/ [1]: https://github.com/anderspitman/fibridge-proxy-rs [2]: https://github.com/omnistreams/omnistreams-spec

In Ask HN: What is your favorite method of sending large files?

anderspitman·May 14, 2020

> In Deno, sockets are still asynchronous, but receiving new data requires users to explicitly read() Interesting. If I understand correctly, they're essentially using pull streams[0]/reactive streams[1]. I compiled a few resources on this topic when I was digging into it a while back[2]. I've found the mental model to be very elegant to work with when needing backpressure in asynchronous systems. As for the dependencies-as-URLs, I don't mind it, and may prefer it. I've been experimenting with minimizing my dependencies lately, and vendoring them in git submodules. It's worked fairly well. [0]: https://github.com/pull-stream/pull-stream [1]: https://github.com/reactive-streams/reactive-streams-jvm [2]: https://github.com/omnistreams/omnistreams-spec

In Deno 1.0

anderspitman·Apr 24, 2020

Thanks for the link. I'm actually quite familiar with that article. It's been very useful for me when designing omnistreams. You may find some of the links on the bottom of this page useful: https://github.com/omnistreams/omnistreams-spec

In Cap'n Proto 0.8: Streaming flow control, HTTP-over-RPC, fibers

apitman·Apr 23, 2020

This is awesome. The new streaming stuff is particularly interesting to me, and it's very impressive that you managed to implement it with no protocol changes. I have a couple questions. Please forgive any misconceptions as I've never used capnproto myself, since all the streaming I've done has to work in the browser, and as far as I know capnproto doesn't work over WebSocket or WebRTC transports. But I've long been impressed with and inspired by capnproto and sandstorm. Main question: is there a reason you opted for traditional window flow control a la TCP, as opposed to "request-N" style like in reactive streams[0] (see rsocket[1] for a great implementation)? So with request-N, a server->client stream would look something like this: interface MyInterface { streamingCall @0 (callback :Callback) -> (requester :Requester); interface C

Read the rest

allback { sendChunk @0 (chunk :Data) -> (); } interface Requester { request @0 (n :int) -> (); } } And the server will only ever send as much data as has been requested by the client with requester calls. This results in really elegant flow control that takes into account both the network, and the client's capacity to consume, without the necessity of tracking windows. The receiver simply calls request(1) each time it processes a message. If you want a buffer you can just start with an assumed N=10, 100 etc. I've found this worked really well when implementing omnistreams[2], which is basically a very thin streaming/multiplexing layer for WebSockets, since WS doesn't have any flow control. (fibridge[3] is a good example of it in action). I started with a window-style but once I learned about reactive streams the request model was much easier to reason about for me. [0]: https://github.com/reactive-streams/reactive-streams-jvm [1]: https://github.com/rsocket/rsocket [2]: https://github.com/omnistreams/omnistreams-spec [3]: http://iobio.io/2019/06/12/introducing-fibridge/

In Cap'n Proto 0.8: Streaming flow control, HTTP-over-RPC, fibers

anderspitman·Sep 16, 2019

Disclaimer: I've only looked fairly close at capnproto, RSocket, and gRPC, never used them myself. It depends on what you need to do. What's your use case? My primary constraint is that I need web browser compatibility, ie websockets currently. If capnproto had a solid browser JS implementation I would said use it hands down if it has the features you need. There's some work being done to shoehorn gRPC into the browser[0], but it seems very complicated to me (requires a proxy server IIRC). If you need a robust browser solution today, take a look at RSocket[1]. I think the reason rsocket isn't talked about more is that it comes across as very "enterprisey", but on a technical level it looks very good to me. You may also be interested in my minimalist approach, omnistreams[2]. It basically adds backpressure and multiplexing on top of websoc

Read the rest

kets. We're running that in production at iobio.io, but the API isn't stable yet (would love external input). One big caveat of capnproto vs gRPC is that capnproto doesn't really have built-in "stream" mechanics, ie the concept of making a request that you expect to return an unbounded list of elements. I've mentioned this before and kentonv explained a way it could be mimicked though[3]. [0] https://grpc.io/blog/state-of-grpc-web/ [1] http://rsocket.io/ [2] https://github.com/omnistreams/omnistreams-spec [3] https://news.ycombinator.com/item?id=20041250

In Sandstorm Oasis Is Shutting Down

anderspitman·Jun 1, 2019

Interesting. I do tend to favor protocols that are a bit lower level and more flexible for composing higher-level functionality. Though this does sound pretty complicated to implement using only what capnproto offers right now. Would there be a way to jury rig "request(n)" backpressure as described in reactive streams[0] (also implemented by RSocket[1]) on top of capnproto? That's what I'm using for omnistreams and it's proven very simple to reason implement and reason about. [0] https://github.com/reactive-streams/reactive-streams-jvm [1] http://rsocket.io/ [2] https://github.com/omnistreams/omnistreams-spec

In Why We’re Switching to gRPC