sqlite-zstd

> I think FS-level compression would be a perfect match. Has anyone tried it successfully on large SQLite DBs? I've had decent success with `sqlite-zstd`[0] which is row-level compression but only on small (~10GB) databases. No reason why it couldn't work for bigger DBs though. [0] https://github.com/phiresky/sqlite-zstd

Personally, 10GB. It's a database of all "person pages" on English Wikipedia with some extracted text (infobox and sections) compressed using sqlite-zstd[1]. About 350k rows. (I know this is "small" compared to what SQLite can handle, but it's the biggest I've worked with. Most of my SQLite databases are under 500MB). [1]: https://github.com/phiresky/sqlite-zstd

Have you compared LZ4 to other compression algorithms, zstd for example? ( https://github.com/phiresky/sqlite-zstd ) Given that LiteFS operates at the filesystem layer via FUSE, have you considered it against designs that use built-in features native to some filesystems? For example, I've considered a similar system design based on a single ZFS primary node that streams ZFS snapshots to reader nodes. With some coordination service (e.g. consul) it could still allow for the whole node promotion process.

In combination with SQLite to allow fast random access (query) and compression: https://github.com/phiresky/sqlite-zstd

You can also try sqlite-zstd [1], which is an sqlite extension allows transparent compression of individual rows by automatically training dictionaries based on groups of data rows. Disclaimer: I made it and it's not production ready [1] https://github.com/phiresky/sqlite-zstd

I'm actually working on a sqlite addon right now that allows to transparently compress tables in sqlite using zstd dictionaries and the preliminary results are really promising! One DB I have compresses down from 1GB to 80MB and random SELECTs are faster in the compressed DB when the cache is cold and only somewhat slower when it is hot :) https://github.com/phiresky/sqlite-zstd