sql.js-httpvfs

So the IPFS part is instead of a CDN? Does it replace asset URLs in templates with e.g. cf IPFS gateway URLs with SRI hashes? datasette-lite > "Could this use the SQLite range header trick?" https://github.com/simonw/datasette-lite/issues/28 From xeus-sqlite-kernel > "Loading SQLite databases from a remote URL" https://github.com/jupyterlite/xeus-sqlite-kernel/issues/6#i... re: "Loading partial SQLite databases over HTTP": > sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs > sqlite-wasm-http: https://github.com/mmomtchev/sqlite-wasm-http >> This project is inspired from @phiresky/sql.js-httpvfs but uses the new official SQLite WASM distribution. Datasette creates a JSON API from a SQLite database, has an optional SQL query editor with canned queries, multi DB query support, docs; https://docs.datasette.io/en/stable/sql_queries.

FUSE does local filesystems, and there are also HTTP remote VFS for sqlite. sqlite-wasm-http is based on phiresky/sql.js-httpvfs, "a read-only HTTP-Range-request based virtual file system for SQLite. It allows hosting an SQLite database on a static file hoster and querying that database from the browser without fully downloading it." "but uses the new official SQLite WASM distribution." [1][2] [2] sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs [1] sqlite-wasm-http: https://www.npmjs.com/package/sqlite-wasm-http [3] sqlite3vfshttp: https://github.com/psanford/sqlite3vfshttp HTTP Range requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requ...

I thought about creating a search engine using https://github.com/phiresky/sql.js-httpvfs, commoncrawl and cloudflare R2. But never found the time to start...

I once tried to use sql.js [1] on a static site for full text search. It worked, but the resulting database size for that site was too large for the web, even with things like detail=none and content='' applied, and requiring the user to download a database each time was just no go. (I guess things should work better for sites with less content or those not requiring a trigram tokenizer.) I switched to Pagefind [2] afterwards before finding out a sql.js-httpvfs [3] fork of sql.js that removes exactly the need to fully download a database (with HTTP range requests). I haven't got the chance to test sql.js-httpvfs out though, but it looks pretty sound and could be much more flexible than Pagefind. (Previously discussed at https://news.ycombinator.com/item?id=27016630 .) [1] https://github.com/sql-js/sql.js/ [2] https://pagefind.app/ [3] htt

For search on static sites, I quite like lunrjs to provide a fast, good enough, client-side-only search experience. I came for the feature set and stayed for the tagline... "Lunr: A bit like Solr, but smaller and not as bright." Another neat trick is to create an FTS search index in SQLite, and have the client query it with sql.js-httpvfs [1]. Works with github pages just fine [2]... Your web server must respect HTTP range requests. [1] https://github.com/phiresky/sql.js-httpvfs [2] With the caveat that you have to commit your database index (lunr's json, or sqlite's db file) to source, because gh pages does not support serving objects managed by git LFS.

You could use a sqlite database and use range queries using something like this: https://github.com/psanford/sqlite3vfshttp https://github.com/phiresky/sql.js-httpvfs Simon Willison wrote about it: https://simonwillison.net/2022/Aug/10/sqlite-http/

Would this implement indexing strategies like HNSW? Linear scan is obviously great to start out with, and can definitely be performant, especially if your data is in a reasonable order and under (say) 10MB, so this shouldn't block a beta release. Also do you build with sqlite-httpvfs? This would go together great https://github.com/phiresky/sql.js-httpvfs