bull

Something like Bull might do the job (https://github.com/OptimalBits/bull)

I have never used Kafka or had to maintain queues with more than a hundred jobs per hour. Most of the jobs queues I've had to maintain are doing simple things like sending emails or updating a third party API. I've always resorted to simpler distributed redis based job queues such as kue [0], bull [1], celery [2] or resque [3]. I would like to learn more about distributed queues and architectures from first principles. Where is a good place to go? [0] https://github.com/Automattic/kue [1] https://github.com/OptimalBits/bull [2] http://www.celeryproject.org/ [3] https://github.com/resque/resque

My thinking here is that had the original devs had a better grasp on redis and postgres, they never would have tried using Kafka in the first place. I can't imagine the problem ever requiring the throughput of Kafka, and there would likely be several other scaling issues in the way of utilizing Kafka to its full potential anyway. I'm pretty sure a redis-based queue like Bull (https://github.com/OptimalBits/bull) would have sufficed for queuing message responses directly on the server (or multiple instances of the server), and while Kafka works fine for long term storage of logs, our use case for the data makes it so it would be far better stored directly in postgres. Postgres is apparently also a decent pub/sub solution, though I'm not sure if it's superior to Kafka in this case. The worst part is that the alternative architecture using a

Haven't used it, but "bull" shows transcoding as their example, so I assume they've run through your use case: https://github.com/OptimalBits/bull