writable-consumable-stream

https://socketcluster.io/ has had such stream implementation and backpressure management since at least 2019. Here's the WritableConsumableStream module: https://github.com/SocketCluster/writable-consumable-stream SocketCluster solves the problem of maintaining message order with async processing. This feature is even more useful now with LLMs as you can process data live, transform streams with AI with no risk of mangling the message order. I may have been the first person to use a for-await-of loop in this way with backpressure. At least on an open source project.

I wrote a library a while back which implements this. It handles all the edge cases, provides the ability to measure message backpressure and allows you to close or abruptly kill a stream: https://github.com/SocketCluster/writable-consumable-stream?... It has full test coverage and is a core part SocketCluster which has almost full e2e test coverage. To put it in perspective, the library itself is very lightweight with under 300 lines of code including dependencies (only 1 of them) and has 1.6K lines of test code... It supports concurrent consumption/iteration by default (each consuming loop causes a new async iterator to be created implicitly in accordance with the AsyncIterator spec) but it also gives you the ability to manually create consumer streams using `stream.createConsumer()` for cases where you want to start buffering messages

Callbacks (including those passed to Promise then methods) are a dangerous primitive. I would recommend using awaitable streams like those used in https://socketcluster.io/ - Or https://github.com/SocketCluster/writable-consumable-stream for general purpose. The philosophy of SC and WritableConsumableStream is that the main trunk logic of your system/module should declared as a single serial stream and any parallel logic should explicitly branch off from it. With traditional event listeners, there is no such thing as a 'main trunk' of the logic... The entry points (event listeners) are scattered all over the place and this can cause a lot of possible state permutations when business logic is executed inside those listeners (since you can't control the timing of when those event handlers are called). This kind of programming is a good alte