hatchet

AI workloads. Stack: Postgres, Go, Typescript, React, Kubernetes Applying: I'm trying something different this time. `ssh hatchet-jobs.com` Note that we're fully open-source, which you can check out here: https://github.com/hatchet-dev/hatchet (and if you have thoughts / opinions / questions about the codebase, please include those in your note!)

ues with data merging, backups and recovery For the OP, I looked at the repo (https://github.com/hatchet-dev/hatchet/blob/main/sql/schema/...) , 1. seems like the schema has database functions - Thats a potential scaling issue, plus you are asking vertical only scalable component to do something which could have taken care by horizontally scalable component 2. TEXT datatype for pretty much every attribute - this is a footgun, you cant use them for indexes properly, in the absence of length checks they can be abused from client side

orkloads. Stack: Postgres, Go, Typescript, React, Kubernetes Applying: email me at alexander@hatchet.run and tell me about something impressive you've built, along with your CV and why you're interested in Hatchet. Note that we're fully open-source, which you can check out here: https://github.com/hatchet-dev/hatchet (and if you have thoughts / opinions / questions about the codebase, please include those in your note!)

ution engine that supports basic pub/sub, like Hatchet). I wrote about our perspective on this here: https://hatchet.run/blog/durable-execution ( disclaimer - I'm one of the people behind https://github.com/hatchet-dev/hatchet )

et [dot] run, and include the following: 1. Tell us about something impressive you've built. 2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it. Or apply online here: https://www.ycombinator.com/companies/hatchet-run/jobs If you don't think you are a good fit for one of the roles, but you're interested in developer tools or infrastructure, please reach out anyway - we'll still consider your application!

lready doing. put shortly - pickaxe brings orchestration and best practices, but you're free to implement to your requirements. [1] https://github.com/hatchet-dev/hatchet

ub.com/hatchet-dev/hatchet/blob/main/sql/schema/... vs https://github.com/hatchet-dev/hatchet/blob/main/sql/schema/...) So to answer the question -- you can query both active queues and a full history of queued tasks up to your retention period, and we've optimized the separate tables for the two different query patterns.

PDATE SKIP LOCKED doesn’t cut it at this scale. After provisioning the largest instance type that CloudSQL offers, we even discussed potentially moving some load off of Postgres in favor of something trendy like Clickhouse + Kafka. But we doubled down on Postgres, and spent about 6 months learning how to operate Postgres databases at scale and reading the Postgres manual and several other resources [0] during commutes and at night. We stuck with Postgres for two reasons: 1. We wanted to make Hatchet as portable and easy to administer as possible, and felt that implementing our own storage engine specifically on Hatchet Cloud would be disingenuous at best, and in the worst case, would take our focus away from the open source community. 2. More importantly, Postgres is general-purpose, which is what makes it both great but hard to scale for some types of workloads. This is also what allows us to offer a general-purpose orchestration platform — we heavily utilize Postgres features like transactions, SKIP LOCKED, recursive queries, triggers, COPY FROM, and much more. Which brings us to today. We’re announcing a full rewrite of the Hatchet engine — still built on Postgres — together with our task orchestration layer which is built on top of our underlying queue. To be more specific, we’re launching: 1. DAG-based workflows that support a much wider array of conditions, including sleep conditions, event-based triggering, and conditional execution based on parent output data [1]. 2. Durable execution — durable execution refers to a function’s ability to recover from failure by caching intermediate results and automatically replaying them on a retry. We call a function with this ability a durable task. We also support durable sleep and durable events, which you can read more about here [2] 3. Queue features such as key-based concurrency queues (for implementing fair queueing), rate limiting, sticky assignment, and worker affinity. 4. Improved performance across every dimension we’ve tested, which we attribute to six improvements to the Hatchet architecture: range-based partitioning of time series tables, hash-based partitioning of task events (for updating task statuses), separating our monitoring tables from our queue, buffered reads and writes, switching all high-volume tables to use identity columns, and aggressive use of Postgres triggers. We've also removed RabbitMQ as a required dependency for self-hosting. We'd greatly appreciate any feedback you have and hope you get the chance to try out Hatchet. [0] https://www.postgresql.org/docs/ [1] https://docs.hatchet.run/home/conditional-workflows [2] https://docs.hatchet.run/home/durable-execution

so an ideal external orchestrator would be ~2ms of latency here. There are also some tradeoffs to a library-only mode that aren't discussed. How would work that requires global coordination between workers behave in this model? Let's say, for example, a global rate limit -- you'd ideally want to avoid contention on rate limit rows, assuming they're stored in Postgres, but each worker attempting to acquire a rate limit simultaneously would slow down start time significantly (and place additional load on the DB). Whereas with a single external orchestrator (or leader election), you can significantly increase throughput by acquiring rate limits as part of a push-based assignment process. The same problem of coordination arises if many workers are competing for the same work -- for example if a machine crashes while doing work, as described in the article. I'm assuming there's some kind of polling happening which uses FOR UPDATE SKIP LOCKED, which concerns me as you start to scale up the number of workers.

ell us about something impressive you've built. 2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.

hub.com/hatchet-dev/hatchet/discussions/436. [1] https://docs.hatchet.run/home/features/triggering-runs/event... [2] https://docs.hatchet.run/sdks/typescript-sdk [3] https://docs.hatchet.run/sdks/python-sdk [4] https://docs.hatchet.run/sdks/go-sdk [5] https://docs.hatchet.run/home/features/rate-limits [6] https://docs.hatchet.run/home/features/concurrency/round-rob... [7] https://docs.hatchet.run/home/features/webhooks [8] https://docs.hatchet.run/home/features/child-workflows