go-sqlite3

My Go driver adds support for this by default, and I provide a simple API to enable it. I'm of the opinion that the VFS shim should ship by default, certainly on the CLI, because it still needs to be enabled for each DB, and all it takes is one instance that doesn't know about checksums to modify a checksummed DB for it to appear corrupt. https://github.com/ncruces/go-sqlite3/tree/main/vfs#checksum... Weirdly, databases created with the macOS version of the sqlite3 CLI reserve 12 bytes, for some reason, so you can't ever enable checksums on those DBs.

I'm the developer of the Wasm driver mentioned in the article: https://github.com/ncruces/go-sqlite3 The v0.20.x series (I just released v0.20.1) should improve this significantly. The driver was reserving lots of address space (with a protected, private, anonymous mapping) which doesn't play well with VMs and OSes that don't like to overcommit. It now reserves much less address space by default, and this can be reduced further. I also brought SQLite WAL mode support to 32-bit archs, to Windows and (through a build tag) to any OS that with an atomic mkdir (used for locking). Happy to discuss here, if you're looking for more in depth commentary.

It gets even more amazing than that. Look at this bit from my GitHub Actions: https://github.com/ncruces/go-sqlite3/blob/fefee692dbfad39f6... I install QEMU (I have the same setup locally), then it's one line each to run unit tests for: Linux 386, arm64, riscv64, ppc64le and s390x. With QEMU installed, all you have to do is: GOARCH=bla go test ./...

I… really don't get this. The people on SQLite, employed to work on this full time for over a year, have this to say (on the link I posted above): “Because the WASM build does not have shared memory APIs, activating WAL requires that a client specifically activate exclusive-locking mode for a db handle immediately after opening it, before doing anything else with it… “WAL mode does not provide any concurrency benefits in this environment. On the contrary, the requirement for exclusive locking eliminates all concurrency support…” I personally worked to implement shared memory WAL for a server side Wasm port of SQLite. But random internet poster decides to “strongly expect it closer to a weekend or two of hacking.” Please, do me a favor and do spend that weekend or two for the benefit of the rest of us all. It'll sincerely be much appreciat

For server side, you'll likely need a different build of Wasm SQLite, that handles concurrency (and file locking) differently. Also, WASI is very far from answer (so far). The SQLite amalgamation builds fine for WASI but concurrency is an unsolved issue. I had to build a VFS from scratch to get my Wasm based SQLite driver into a usable shape. https://github.com/ncruces/go-sqlite3/blob/main/vfs/README.m...

For use in Golang, I really like ncruces wasm SQLite package - https://github.com/ncruces/go-sqlite3 . Unlike cznic's go package (which is great, btw), the wasm version works well on OpenBSD and the like.

If you decide to use SQLite with one (single thread) writer pool and an other reader pool, this may help: https://github.com/bxcodec/dbresolver Also sometimes if I have two tables where I know I’ll never need to do a JOIN between them, I’ll just put them in separate databases so that I can connect to them independently. If this data belongs together and you're just doing this to improve concurrency, this may be a case where BEGIN CONCURRENT helps: https://sqlite.org/src/doc/begin-concurrent/doc/begin_concur... If you want to experiment with BEGIN CONCURRENT in Go you could do worse than try my SQLite driver: https://github.com/ncruces/go-sqlite3 Import this package to get the version with BEGIN CONCURRENT: https://github.com/ncruces/go-sqlite3/tree/main/embed/bcw2

Has been done and works pretty great: https://github.com/ncruces/go-sqlite3. Though doesn’t convert the WASM to C, it runs the WASM in Wazero instead.

In the specific case of SQLite, you can use it through WASM now [1]. It uses the dependency and Cgo-free Wazero runtime. Performance so far has been better than the modernc transpile and it’s probably sufficient for a lot of use cases. [1] https://github.com/ncruces/go-sqlite3

Yep, I just made it tweakable at build, which was always the intent, although I expect the default to be popular. https://github.com/ncruces/go-sqlite3/blob/67d859a5/vfs/adia... That's unfortunate about the default parameters, but note that you can also replace the KDF altogether (besides just not using it). You just need to implement this interface, with any HBSH construction and KDF: https://github.com/ncruces/go-sqlite3/blob/67d859a5/vfs/adia... If you keep the HBSH and change the KDF, your file format will be “compatible.”

My extension targets my Go SQLite bindings (the VFS is implemented in Go) [1]. This extension, and the wrapper, target public SQLite APIs, so if anyone wants to replace SQLite with libSQL, that should be easy. You simply need to include a little bit of glue C code that I append to the amalgamation, and use some SQLite compile options. I explicitly support building and providing your own, custom, Wasm SQLite "blob" (it just needs to include my glue code). As for Adiantum encryption, as I said, reimplementing this in C/C++/Rust to make it a loadable extension is perfectly viable, and would be compatible with libSQL, again because this uses only public SQLite APIs (that's the point, basically!) But this is predicated on it being a secure scheme (that's the feedback I was looking for this time around). PS: I've got nothing against Turso, or l

SQLite encryption at-rest is a hot requested feature of both the “default” CGo driver [1] and the transpiled alternative driver [2]. So, this is a feature I wanted to bring to my own Wasm based Go driver/bindings [3]. Open-source SQLite encryption extensions have had a troubled last few years. For whatever reason, in 2020 the (undocumented) feature that made it easy to offer page-level encryption was removed [4]. Some solutions are stuck with SQLite 3.31.1, but Ulrich Telle stepped up with a VFS approach [5]. Still, their solution seemed harder than something I'd want to maintain, as it requires understanding the structure of what's being written to disk at the VFS layer. So, I looked at full disk encryption for something with less of an impedance mismatch. Specifically, I'm using the Adiantum tweakable and length-preserving encryption (w

struggling figuring out how to make my cgo sqlite cross-compile to Windows Plenty of people trying to fix that. There's at least: https://modernc.org/sqlite Then there's https://github.com/zombiezen/go-sqlite that actually builds https://crawshaw.io/sqlite on top of modernc. And there's mine that has both a low-level and a database/sql driver and which builds and runs everywhere Go does: https://github.com/ncruces/go-sqlite3