sql.js-httpvfs

Oh that’s an interesting idea. I saw someone built SQLite over HTTP with the Range header: https://phiresky.github.io/blog/2021/hosting-sqlite-database... So presumably in a similar manner as https://github.com/phiresky/sql.js-httpvfs you could map SQLite pages to leaf torrents too and get the chunking you are looking for.

> Also, SQLite's VFS API can do a lot there already. For example, I remember seeing SQLite compiled to WASM, using a VFS that downloads a remote database on S3 using HTTP Range requests. (I don't think it supported writing to the database, but it was still a really cool way of allowing complex client-side querying of a dataset that is static but too big to transfer). <https://github.com/phiresky/sql.js-httpvfs>

I was wondering how this ran queries against SQLite files in an S3 bucket. Here's the answer - it's using some very clever Python ctypes code to implement that HTTP Range header trick from last year: https://github.com/michalc/sqlite-s3-query/blob/main/sqlite_... Here's the inspiration for that: https://github.com/phiresky/sql.js-httpvfs

Seems like a great use case for static sqlite hosting https://github.com/phiresky/sql.js-httpvfs

How close is Python SQLite and Datasette Lite to accessing a hosted SQL database using HTTP range requests as can be done in sql.js like https://github.com/phiresky/sql.js-httpvfs? I put together a Pyodide-based web app where users need a few indexed queries from a 600mb SQLite database but it isn't very practical for them to download the whole thing into the browser. I would datasettify it and access it by API, if I wasn't relying on a pyodide library that needs direct SQLite access. https://observablehq.com/@thadk/life

You can also query sqlite databases directly from an http server that supports range requests (like s3). There are a bunch of implementations of this in different languages including Go[0] and Javascript[1]. [0]: https://github.com/psanford/sqlite3vfshttp [1]: https://github.com/phiresky/sql.js-httpvfs

From "Hosting SQLite Databases on GitHub Pages" https://news.ycombinator.com/item?id=28021766 https://westurner.github.io/hnlog/#comment-28021766 : DuckDB can query [and page] Parquet from GitHub, sql.js-httpvfs, sqltorrent, File System Access API (Chrome only so far; IDK about resource quotas and multi-GB datasets), serverless search with WASM workers https://github.com/phiresky/sql.js-httpvfs : > sql.js is a light wrapper around SQLite compiled with EMScripten for use in the browser (client-side). > This [sql.js-httpvfs] repo is a fork of and wrapper around sql.js to provide a read-only HTTP-Range-request based virtual file system for SQLite. It allows hosting an SQLite database on a static file hoster and querying that database from the browser without fully downloading it. > The virtual file system is an emscripten filesystem with som

Hi HN, this is most likely the most primitive internet search engine of all time, built with CommonCrawl data [1]. It would have been a feasible candidate to work with sql.js-httpvfs [2], a WebAssembly-compiled SQLite driver on a vfs to be able to omit the server part - but when I tried it, the performance with larger result numbers was a bit too slow to be usable, so it wasn't possible. Try searching for your hometown, for example. You'll get quite uncommon yet interesting results. Full of spam though. It really makes you appreciate the enormous filtering that common search engines apply to their results. [1] https://commoncrawl.org/ [2] https://github.com/phiresky/sql.js-httpvfs

It doesn't have to be range requests, you could split the file instead of depending on the range requests. Essentially it's the same as the chunking instructions for hosters who have a maximum file size as is demonstrated here: https://github.com/phiresky/sql.js-httpvfs/blob/master/creat...

Continuing my looking into this, the SQLite range requests trick was implemented in this [0] project. It turns out that project was inspired by a few others [1], two of which [2][3] implement a vfs for SQLite on top of bittorrent doing exactly this suggestion, but with a bittorrent hosted file rather than IPFS. They only download the parts of the SQLite files needed when querying. 0: https://github.com/phiresky/sql.js-httpvfs 1: https://github.com/phiresky/sql.js-httpvfs#inspiration 2: https://github.com/lmatteis/torrent-net 3: https://github.com/bittorrent/sqltorrent