cdb64

Another solution for cases where you don't add new entries all the time and can reindex the whole database every once in a while instead: use cdb. There's a nice description of the internals and how it works. http://www.unixuser.org/~euske/doc/cdbinternals/index.html It guarantees access in two disk reads. The original version has the limit of 4gb, but there are 64b versions as well - for example https://github.com/pcarrier/cdb64?files=1

If you're looking for a very straightforward 64-bit port, [I'm the author of] https://github.com/pcarrier/cdb64

We ended up using sparkey as the first backend for hammerspace, but hammerspace was written to support multiple backends. We benchmarked both cdb and sparkey, and their performance was very similar for our use case. At under 100mb of data, we weren't concerned about the 4G limitation or about the data not fitting in cache. I don't think sparkey has the 4G limitation, and someone has already forked cdb to support 64 bit offsets: https://github.com/pcarrier/cdb64

A 64-bit port of cdb isn't too hard: https://github.com/pcarrier/cdb64