verneuil

That's how https://github.com/backtrace-labs/verneuil 's read replication is meant to be used. There's a command-line tool to recreate a sqlite DB file from a snapshot's manifest, with an optional local cache to avoid fetching unchanged pages, or you can directly use a replica in memory, with pragmas for (async) refreshes. The write tracking needs to intercept all writes with a custom VFS, but once registered and configured, it's regular in-memory SQLite (no additional daemon).

Very cool, I'd have a big use case for "Litestream in Rust" as a library. Any way to get notified about the release? Also: will this be a straight port, or could it also support "live-replication" from S3, instead of just on-demand restores? https://github.com/backtrace-labs/verneuil supports this. Sadly it isn't very actively maintained and buggy.

Verneuil https://github.com/backtrace-labs/verneuil might work for you. Its replication client can use a persistent chunk cache (no fsync, so doesn't survive reboots), and there's a CLI utility to reconstitute a sqlite db file (`verneuilctl restore`). It also tries to do the right thing by writing to a temporary file before renaming it over the old db file. On the write-side, you can load the verneuil VFS as a runtime extension. There's a similar gotcha with reboots, where the replication state is persisted but not fsync-ed, so you get a full refresh on reboots. The S3 storage is content addressed though, so you "only" pay for useless bandwidth and API calls, not storage.

https://github.com/backtrace-labs/verneuil doesn't even need an HTTP connection between the writer and readers: readers only interact with the object store.

Nifty, reminds me of https://github.com/backtrace-labs/verneuil or possibly https://litestream.io/

Verneuil (https://github.com/backtrace-labs/verneuil) offers that for read replicas, because backtrace as well has some multi-GB sqlite DBs. It's a VFS (loadable at runtime as a sqlite extension), instead of a filesystem. I don't remember if that's in the docs; commit that adds the configuration field https://github.com/backtrace-labs/verneuil/commit/027318ba74... and commit for the enum https://github.com/backtrace-labs/verneuil/blob/e6697498f3ba... (the actual implementation is nothing special https://github.com/backtrace-labs/verneuil/commit/b6bdfcf7bc...) Issue for the feature: https://github.com/backtrace-labs/verneuil/issues/12

https://github.com/backtrace-labs/verneuil is in-process (precisely to minimise deployment pain).

https://github.com/backtrace-labs/verneuil/ is one way to address the diffing / read replica part of the problem. I believe it's compatible with gossipping: most of the data is in small content-addressed chunks, with small manifests that tell clients what chunks to fetch and how to reassemble them to recreate a sqlite database. There's already client-side caching to persistent storage, and chunks can be fetched on demand. Sharing replication data P2P, while retaining the simplicity of a single authoritative writer per database, is explicitly part of the project's long-term goals!