arniesmtpbufferserver

You don't even need a database to make a message queue. The Linux file system makes a perfectly good basis for a message queue since file moves are atomic. My guess is that many people are implementing queuing mechanisms just for sending email. You can see how this works in Arnie SMTP buffer server, a super simple queue just for emails, no database at all, just the file system. https://github.com/bootrino/arniesmtpbufferserver

Any time you see "atomic", you can make a queue. A practical application of atomic mv is building simple file based queuing mechanisms. For example I wrote this SMTP buffer server which moves things to different directories as a simple form of message queue. https://github.com/bootrino/arniesmtpbufferserver It's pretty simple - ~100 lines of Python without comments etc. https://raw.githubusercontent.com/bootrino/arniesmtpbufferse... Caveat I think this needs examination from the perspective of fsync - i.e. I suspect the code should be fsyncing at certain points but not sure. I also wrote (in Rust) a simple file based message queue using atomic mv. It instantly maxed out the SSD performance at about 30,000 messages/second.

I wrote a message queue in Python called StarQueue. It’s meant to be a simpler reimagining of Amazon SQS. It has an HTTP API and behaves mostly like SQS. I wrote it to support Postgres, Microsoft’s SQL server and also MySQL because they all support SKIP LOCKED. At some point I turned it into a hosted service and only maintained the Postgres implementation though the MySQL and SQL server code is still in there. It’s not an active project but the code is at https://github.com/starqueue/starqueue/ After that I wanted to write the worlds fastest message queue so I implemented an HTTP message queue in Rust. It maxed out the disk at about 50,000 messages a second I vaguely recall, so I switched to purely memory only and in the biggest EC2 instance I could run it on it did about 7 million messages a second. That was just a crappy prototype so I

Not just Postgres. You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED. Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible. You could write a file system based message queue in 100 lines of async python, which I did here: https://github.com/bootrino/arniesmtpbufferserver File system based message queues can be written in any language, extremely simple and, most importantly - zero configuration. One of the most frustrating things about queuing systems is configuration - that includes database backed queuing systems. They can also be fast - I wrote one in Rust which maxed out the hard disk's rando

I suspect in many cases people use Celery only to send emails or for simple queueing type problems. At risk of being overly negative, my experience with celery has been that it is complex and hard to configure and debug. I chose a simple queueing mechanism instead of using Celery, and for sending emails I was so frustrated by Celery that I wrote the Arnie SMTP buffer server (in 100 lines of Python): https://github.com/bootrino/arniesmtpbufferserver

If you are just queueing outbound emails you can consider a simple SMTP buffer that uses no server at all - it just saves files to disk. I was using Celery for sending emails - nothing else. And Celery was such a nightmare to configure and debug and such overkill for email buffering that in a fit of frustration I wrote the Arnie SMTP buffering server and ditched Celery. https://github.com/bootrino/arniesmtpbufferserver It's only 100 lines of code: https://github.com/bootrino/arniesmtpbufferserver/blob/maste...

Here's the 100 lines of code: https://github.com/bootrino/arniesmtpbufferserver/blob/master/arniesmtpbufferserve… Here's the github repo: https://github.com/bootrino/arniesmtpbufferserver It's MIT licensed. Arnie is a server that has the single purpose of buffering outbound SMTP emails. I wrote Arnie because I got frustrated using complex queueing systems simply to buffer outbound SMTP emails. I'd rather spend time writing this than spend the same time debugging Celery configuration. A typical web SAAS needs to send emails such as signup/signin/forgot password etc. The web page code itself should not directly write this to an SMTP server. Instead they should be decoupled. There's a few reasons for this. One is, if there is an error in sending the email, then the whole thing simply falls over if that send was executed by the web page co