sqlite_zstd_vfs

I've never done it, but for the readonly case, I think all you need to is to combine the existing reader VFS with zstd-seekable-format-go. https://github.com/ncruces/go-sqlite3/discussions/117#discus... For read/write, I'm honestly not sure. The Zipvfs is an… erm… architectural mess that only really works because it accesses private SQLite APIs. Which is fine, but history has shown the SQLite team is willing to break those APIs, as they did for the ones they build the SQLite encryption extension on. https://sqlite.org/zipvfs/doc/trunk/www/howitworks.wiki The zstandard alternative is sqlite_zstd_vfs, which faces the same architectural issues. So, I'd rather not go there. But should be doable, as long as you're not needing private APIs. https://github.com/mlin/sqlite_zstd_vfs

I did actually previously write a VFS, though it did something else entirely: https://phiresky.github.io/blog/2021/hosting-sqlite-database... You're right that a comparison to compression VFS is completely missing. I knew about their existence but I have to admit I didn't look too closely. Note that the main "novelty" here is training a shared / partitioned compression dictionary specifically to take advantage of redundancy that wouldn't appear within only one row or even within a database page / block . The compression happens at the row level and can additionally use application knowledge - you couldn't do that in the VFS level. For example, you can have separate compression dictionaries per different columns and per groups of rows with some commonality. I'll have to compare to a compression vfs (do you have a favorite?) and see if mayb

I plowed the 40 GiB database file into sqlite_zstd_vfs [1] which reduced it 75% to 10 GiB. This plug-in also supports HTTP access [2] in the spirit of phiresky's, however, I don't have a WebAssembly build of mine yet, so it's a library for desktop/command-line apps for now. You can try it out on Linux or macOS x86-64: pip3 install genomicsqlite genomicsqlite https://f000.backblazeb2.com/file/mlin-public/static.wiki/en.zstd.db "select text from wiki_articles where title = 'SQLite'" ("genomicsqlite" is the CLI for my Genomics Extension [3], which is built around these Zstandard compression & web layers.) [1] https://github.com/mlin/sqlite_zstd_vfs [2] https://github.com/mlin/sqlite_web_vfs [3] https://mlin.github.io/GenomicSQLite EDITS: I expanded on this comment in this gist https://gist.github.com/mlin/ee20d7c5156baf9b12518961f36590c