sqlightning

Fun footnote: SQLite only got on board with mmap after I demonstrated how slow their code was without it. I.e., getting a 22x speedup by replacing SQLite's btree code with LMDB https://github.com/LMDB/sqlightning

https://github.com/LMDB/sqlightning https://github.com/LumoSQL/LumoSQL

I'm guessing you mean "and the software is fast enough", because OpenLDAP itself has been the world's fastest distributed database for over a decade, and nothing else even comes close. Since you built on top of SQLite, then you're already at least an order of magnitude slower. You could speed it up a bit by using SQLightning, which replaces SQLite's Btree engine with OpenLDAP's. https://github.com/LMDB/sqlightning But yes, it's obvious your aim is to provide bare minimum LDAP functionality for people who were only taught SQL and have no desire to learn LDAP. A bit of a loss, because the workloads LDAP is used for most often are things RDBMSs suck at.

Have you seen SQLightning and/or LumoSQL? * https://github.com/LMDB/sqlightning * https://github.com/LumoSQL/LumoSQL SQLightning was the initial project combining SQLite3 with an LMDB backend. It seemed to be more an experimental/Proof-of-Concept thing, and isn't maintained. LumoSQL is an alternative project (maintained), providing a SQLite3 front end with various optional storage backends. One of which is LMDB. Note - I'm not affiliated with either project, I just remembered they exist. :)

This has been shared without context but I guess the SQLite team is starting to modularize the btree code in order to facilitate work like SQLightning: https://github.com/LMDB/sqlightning At the time SQLightning greatly improved SQLite performance but due to LMDB's requirement to have keys fit in 2/3 of a page it wasn't really useful as a general purpose replacement of SQLite's internal b-tree implementation. EDIT: It looks like SQLightning got adopted and has been worked on by the SQLite team under the name LumoSQL. Here's the project's readme: https://lumosql.org/src/lumosql/doc/trunk/README.md which contains at the end "A Brief History of LumoSQL"

LMDB and SQLite are not directly comparable. LMDB is a transactional B+tree-based key/value store. SQLite is an implementation of a transactional SQL data model on top of a B+tree-based key/value store, so it is logically at least one abstraction layer higher than LMDB. (Key/value stores underlie pretty much all of the other data models you'll ever use.) That aside - LMDB is not just smaller, faster, and more reliable than SQLite, it is also smaller/faster/more reliable than SQLite's own B+tree implementation, and SQLite can be patched to use LMDB instead of its own B+tree code, resulting in a smaller/faster footprint for SQLite itself. Proof of concept was done here https://github.com/LMDB/sqlightning A new team has picked this up and carried it forward https://github.com/LumoSQL/LumoSQL Generally, unless your application has fairly simp

Can this be made into a back end for sqlite? edit example like this: https://github.com/LMDB/sqlightning

LMDB is a storage engine whereas SQLite is a small database. There is even a version of SQLite that used LMDB as the underlying storage engine: https://github.com/LMDB/sqlightning.

We have only ever compared LMDB in synchronous mode to other DBs in synchronous mode, and LMDB in asynch mode to other DBs in asynch mode. Come on, that's too obvious. And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/ Replacing SQLite's Btree engine with LMDB makes the SQLite footprint smaller, faster, and more reliable too. https://github.com/LMDB/sqlightning

The Btree engine in SQLite is quite inefficient, and also unreliable on its own. (Needs the transaction log support to provide reliability.) LMDB blows it away. https://github.com/LMDB/sqlightning

I wonder if they got the inspiration from https://github.com/LMDB/sqlightning

I'm surprised there wasn't more investigation of SQLite and LMDB: https://github.com/LMDB/sqlightning The performance there shows either little to no performance difference, up to substantial speed increases.

Premature optimization is evil, but preemptive optimization is necessary unless you want to paint yourself into a corner. I realized this after implementing a bitcoin full node. In my bitcoin implementation, as an experiment, I tried storing the blockchain in sqlite, postgres, and leveldb. I gathered up a bunch of data from the first ~200k blocks of the blockchain and benchmarked all three databases. I queried for something like 30,000 utxos out of a set of a couple million. What took 300-400ms in leveldb took 1.6 seconds in postgres (on the repl. in my actual node it would have taken longer due to deserialization of the utxos). What took 1.6 seconds in postgres took over 30 seconds in SQlite. Now, you can tell me I did the benchmarks wrong, and "oh, if you just did this it would be faster!", but 30+ seconds is slower to an absolutely ins