sql.js-httpvfs

There is also the deep magic... https://github.com/phiresky/sql.js-httpvfs

if performance becomes a problem statically hosting sqlite DBs with client side queries and http range requests is an interesting approach: https://github.com/phiresky/sql.js-httpvfs

Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqlite.gz https://hackerbook.dosaygo.com/static-shards/shard_1635.sqlite.gz https://hackerbook.dosaygo.com/static-shards/shard_1634.sqlite.gz As I paginated to previous days. It's reminiscent of that brilliant SQLite.js VFS trick from a few years ago: https://github.com/phiresky/sql.js-httpvfs - only that one used HTTP range headers, this one uses sharded files instead. The interactive SQL query interface at https://hackerb

correct! it's a sqlite database which ends up being automatically gzip'd by the github cdn aka fastly - this ends up being ~4mb over the wire obviously this solution doesn't scale (versus something like https://github.com/phiresky/sql.js-httpvfs ) but sometimes it's fun to "just" send the sqlite database to the frontend and be done :-)

From "Show HN: TeaTime – distributed book library powered by SQLite, IPFS and GitHub" https://news.ycombinator.com/item?id=42264274 : >> phiresky/sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs >> mmomtchev/sqlite-wasm-http: https://github.com/mmomtchev/sqlite-wasm-http >> This project is inspired from @phiresky/sql.js-httpvfs but uses the new official SQLite WASM distribution duckdb/duckdb-wasm: https://github.com/duckdb/duckdb-wasm

There are implementations for that: For example https://github.com/psanford/sqlite3vfshttp or https://github.com/phiresky/sql.js-httpvfs

So the IPFS part is instead of a CDN? Does it replace asset URLs in templates with e.g. cf IPFS gateway URLs with SRI hashes? datasette-lite > "Could this use the SQLite range header trick?" https://github.com/simonw/datasette-lite/issues/28 From xeus-sqlite-kernel > "Loading SQLite databases from a remote URL" https://github.com/jupyterlite/xeus-sqlite-kernel/issues/6#i... re: "Loading partial SQLite databases over HTTP": > sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs > sqlite-wasm-http: https://github.com/mmomtchev/sqlite-wasm-http >> This project is inspired from @phiresky/sql.js-httpvfs but uses the new official SQLite WASM distribution. Datasette creates a JSON API from a SQLite database, has an optional SQL query editor with canned queries, multi DB query support, docs; https://docs.datasette.io/en/stable/sql_queries.

FUSE does local filesystems, and there are also HTTP remote VFS for sqlite. sqlite-wasm-http is based on phiresky/sql.js-httpvfs, "a read-only HTTP-Range-request based virtual file system for SQLite. It allows hosting an SQLite database on a static file hoster and querying that database from the browser without fully downloading it." "but uses the new official SQLite WASM distribution." [1][2] [2] sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs [1] sqlite-wasm-http: https://www.npmjs.com/package/sqlite-wasm-http [3] sqlite3vfshttp: https://github.com/psanford/sqlite3vfshttp HTTP Range requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requ...

I thought about creating a search engine using https://github.com/phiresky/sql.js-httpvfs, commoncrawl and cloudflare R2. But never found the time to start...

I once tried to use sql.js [1] on a static site for full text search. It worked, but the resulting database size for that site was too large for the web, even with things like detail=none and content='' applied, and requiring the user to download a database each time was just no go. (I guess things should work better for sites with less content or those not requiring a trigram tokenizer.) I switched to Pagefind [2] afterwards before finding out a sql.js-httpvfs [3] fork of sql.js that removes exactly the need to fully download a database (with HTTP range requests). I haven't got the chance to test sql.js-httpvfs out though, but it looks pretty sound and could be much more flexible than Pagefind. (Previously discussed at https://news.ycombinator.com/item?id=27016630 .) [1] https://github.com/sql-js/sql.js/ [2] https://pagefind.app/ [3] htt

For search on static sites, I quite like lunrjs to provide a fast, good enough, client-side-only search experience. I came for the feature set and stayed for the tagline... "Lunr: A bit like Solr, but smaller and not as bright." Another neat trick is to create an FTS search index in SQLite, and have the client query it with sql.js-httpvfs [1]. Works with github pages just fine [2]... Your web server must respect HTTP range requests. [1] https://github.com/phiresky/sql.js-httpvfs [2] With the caveat that you have to commit your database index (lunr's json, or sqlite's db file) to source, because gh pages does not support serving objects managed by git LFS.

You could use a sqlite database and use range queries using something like this: https://github.com/psanford/sqlite3vfshttp https://github.com/phiresky/sql.js-httpvfs Simon Willison wrote about it: https://simonwillison.net/2022/Aug/10/sqlite-http/

Would this implement indexing strategies like HNSW? Linear scan is obviously great to start out with, and can definitely be performant, especially if your data is in a reasonable order and under (say) 10MB, so this shouldn't block a beta release. Also do you build with sqlite-httpvfs? This would go together great https://github.com/phiresky/sql.js-httpvfs

I couldn't find do these wrappers support read-only SQLite databases with HTTP range requests, like in this famous post [1]. Phiresky's wrapper supports it, but it seems to be rebuilding the whole sql.js [2], I'd rather have it as VFS on top of sqlite.org's own WASM module. I like the idea of HTTP range requests, but I don't want to run a fork, that will be unmaintained in few years. [1]: https://news.ycombinator.com/item?id=27016630 [2]: https://github.com/phiresky/sql.js-httpvfs

I used this[0] awesome library to do just this over http. I posted more info in my other comment. I'm still exploring but so far it's been pretty impressive to me. [0] https://github.com/phiresky/sql.js-httpvfs