flow

At Estuary, we have an in-house Rust crate [1] for building scale-out durable actors / FSMs in Postgres. It powers all async activity in our control plane -- slews of fine-grain scheduled actions, complex change propagation through data-flow topologies, reliable alert and email delivery, and more -- at hundreds to thousands of state transitions per second (today). It's been a wonderful pattern to build on, and is all of three source files. Here's a an example computing a Fibonacci sequence (very inefficiently, with lots of spawned sub-tasks and message passing) [2] [1] https://github.com/estuary/flow/tree/master/crates/automatio... [2] https://github.com/estuary/flow/blob/master/crates/automatio...

The key ingredient is leveled reads across topics. A task which is transforming across multiple joined topics, or is materializing multiple topics to an external system, needs to read across those topics in a coordinated order so that _doesn't_ happen. Minimally by using wall-clock publication time of the events so that "A before B" relationships are preserved, and ideally using transaction metadata captured from the source system(s) so that transaction boundaries propagate through the data flow. Essentially, you have to do a streaming data shuffle where you're holding back some records to let other records catch up. We've implemented this in our own platform [1] (I'm co-founder). You can also attenuate this further by deliberately holding back data from certain streams. This is handy when stream processing because you sometimes want to r

We use a macaddr8 that embeds a wall-clock timestamp (so they're ascending order, achieving data locality) with some additional shard and sequence-number bits. It's worked really well for us: https://github.com/estuary/flow/blob/master/supabase/migrati... we use macaddr8 instead of bigint, because it has a postgres serialization / JSON encoding which lossless-ly round-trips with browsers and it works well with PostgREST. The same CANNOT be said for bigint, which is a huge footgun.

Estuary Technology | Backend Engineer | Developer Evangelist | Rust, Go | REMOTE OR HYBRID | UTC-7 to UTC+2 Regional offices in NYC & Columbus, OH Estuary (https://www.estuary.dev/) is the first real-time Data Operations platform for future-proof pipelines, including both historical and real-time data set up in minutes. Our team is rapidly growing, VC funded and led by two successful, repeat founders. We primarily develop in Rust and Go and are heavily built on top of gazette which is an internally developed streaming engine. Flow: https://github.com/estuary/flow Gazette: https://gazette.readthedocs.io/en/latest/ Backend Engineer: https://www.estuary.dev/about/#backend Developer Evangelist: https://www.estuary.dev/about/#developerevangelist ^This is an exciting opportunity to make direct impact and shape user perception of a new product t

Estuary Technology | Backend Engineer | Rust, Go | REMOTE OR HYBRID | Regional offices in NYC & Columbus, OH Estuary (https://www.estuary.dev/) is the first real-time Data Operations platform for future-proof pipelines, including both historical and real-time data set up in minutes. Our team is rapidly growing, VC funded and led by two successful, repeat founders. We primarily develop in Rust and Go and are heavily built on top of gazette which is an internally developed streaming engine. Flow: https://github.com/estuary/flow Gazette: https://gazette.readthedocs.io/en/latest/ We are looking for a backend engineer who is early in their career (around 1-3 years of industry experience) to join our team. For engineers who have an unquenched curiosity and drive to solve complex distributed systems problems, this is an opportunity to advance yo

Estuary Technology | Junior Backend Engineer | Rust, Go | REMOTE OR HYBRID | Regional offices in NYC & Columbus, OH Estuary (https://www.estuary.dev/) is the first real-time Data Operations platform for future-poof pipelines, including both historical and real-time data set up in minutes. Our team is rapidly growing, VC funded and led by two successful, repeat founders. We primarily develop in rust and go and are heavily built on top of gazette which is an internally developed streaming engine. Flow: https://github.com/estuary/flow Gazette: https://gazette.readthedocs.io/en/latest/ We are looking for a junior backend engineer with 2-3 years of industry experience. For engineers who have an unquenched curiosity and drive to solve complex distributed systems problems, this is an opportunity to advance your career alongside a team of subject

Estuary Technology | Developer Evangelist | Backend Engineer | Rust, Go | REMOTE OR HYBRID | Regional offices in NYC & Columbus, OH Estuary (https://www.estuary.dev/) is an open-source DataOps platform that empowers engineering teams to build real-time, data-intensive applications at scale with minimal friction. We primarily develop in rust and go and are heavily built on top of gazette which is an internally developed streaming engine. Flow: https://github.com/estuary/flow Gazette: https://gazette.readthedocs.io/en/latest/ Our team is rapidly growing, VC funded and led by two successful, repeat founders. Estuary offers full health benefits, competitive salary, unlimited PTO, 401K, equity, and an inclusive, supportive work environment to expand your skills among a passionate team. Open Positions: -Developer Evangelist https://www.estuar

It does seem like a weird bug that this would cause errors https://github.com/estuary/flow/runs/5642694619?check_suite_... seems like it should be some kind of warning instead of an error?

> I'd love to be able to have a "relational database" that is also some "NoSQL" databases (since there's a few major useful paradigms there) that just takes care of this for me. I imagine I'd have to declare my schemas, but I'd love it if that's all I had to do and then the DB handled keeping sync and such. You might be interested in what we're building [0] It synchronizes your data systems so that, for example, you can CDC tables from your Postgres DB, transform them in interesting ways, and then materialize the result in a view within Elastic or DynamoDB that updates continuously and with millisecond latency. It will even propagate your sourced SQL schemas into JSON schemas, and from there to, say, equivalent Elastic Search schema. [0]: https://github.com/estuary/flow

Not for Kafka, but we are building Flow [1] to offer deterministic ordering, even across multiple logical and physical partitions, and regardless of whether you're back-filling over history or processing in real-time. This ends up being required for one of our architectural goals, which is fully repeatable transformations: You must be able to model a transactional decision as a Flow derivation (like "does account X have funds to transfer Y to Z ?", and if you create a _copy_ of that derivation months later, get the exact same result. Under the hood (and simplifying a bit) Flow always does a streaming shuffled read to map events from partitions to task shards, and each shard maintains a min-heap to process events in their ~wall-time order. This also avoids the common "Tyranny of Partitioning", where your upstream partitioning parallelism N

If you're in the Go ecosystem, Gazette [0] offers transactional integrations [1] with remote DB's for stateful processing pipelines, as well as local stores for embedded in-process state management. It also natively stores data as files in cloud storage. Brokers are ephemeral, you don't need to migrate data between them, and you're not constrained by their disk size. Gazette defaults to exactly-once semantics, and has stronger replication guarantees (your R factor is your R factor, period -- no "in sync replicas"). Estuary Flow [2] is building on Gazette as an implementation detail to offer end-to-end integrations with external SaaS & DB's for building real-time dataflows, as a managed service. [0]: https://github.com/gazette/core [1]: https://gazette.readthedocs.io/en/latest/consumers-concepts.... [2]: https://github.com/estuary/flow

Estuary Flow [1] may be interesting to those in this space. We're still building, but it's a GitOps workflow tool that tightly integrates schema definition (JSON Schema), captures and materializations from/to your systems & SaaS, rich transformations, catalog and provenance metadata tracking, built-in testing, and a managed runtime. All with sub-second latency. Flow's runtime uses nascent but really promising open protocols for building connectors to the myriad systems and APIs out there. We're seeing Airbyte's work (itself built off of Singer) as the best steps in this direction and are leaning into that effort ourselves. [1] github.com/estuary/flow

Can you speak more to how updates get pushed out to your UI clients? Are you using Firestore ? Hasura? Polling? Something we're looking at enabling w/ Estuary Flow [1] is materializing to Firestore for this use case (and Flow already supports PostgreSQL materializations). Flow would capture changes from your database tables (or you write normalized events directly to Flow collections). You aggregate and join as needed, and then materialize your views into Firestore (or another Postgres table) where you'd subscribe for live updates. [1]: https://github.com/estuary/flow