klevdb

When I adopted mmap in klevdb [1], I saw a dramatic performance improvements. So, even as klevdb completes a write segment, it will reopen, on demand, the segment for reading with mmap (segments are basically part of write only log). With this any random reads are super fast (but of course not as fast as sequential ones). [1] https://github.com/klev-dev/klevdb

Interesting, the performance of https://github.com/klev-dev/klevdb is 10x https://github.com/maragudk/goqite so it makes me assume the durability is somewhat lacking. Can you speak to the tradeoffs here around message loss? I would think that having a small chance of message loss due to writing to an append only log in batches might be a reasonable trade off for many things (if that is how it works).

In a me too way, I also created something similar - https://github.com/klev-dev/klevdb It started on top of bbolt, but over time moved a completely custom implementation. The main reason for this was performance, but I also wanted to run a lot of queues, so each one had to be as light as possible. On top it even provides some indexing capabilities, which (when enabled) allow for quick find by key and time.

https://klev.dev I'm fascinated by the idea of Kafka and wanted to use it like a saas in my own apps, so I made my own take on it. It also doubles as a key/value store, so its useful for a bunch of things. The store itself is OSS and you can find it at https://github.com/klev-dev/klevdb.