torrent-net

Continuing my looking into this, the SQLite range requests trick was implemented in this [0] project. It turns out that project was inspired by a few others [1], two of which [2][3] implement a vfs for SQLite on top of bittorrent doing exactly this suggestion, but with a bittorrent hosted file rather than IPFS. They only download the parts of the SQLite files needed when querying. 0: https://github.com/phiresky/sql.js-httpvfs 1: https://github.com/phiresky/sql.js-httpvfs#inspiration 2: https://github.com/lmatteis/torrent-net 3: https://github.com/bittorrent/sqltorrent

I believe this is the project mentioned https://github.com/lmatteis/torrent-net

A torrent client that exposes torrent content as files: https://github.com/distribyted/distribyted It's pretty fun to work on it and implement new use cases. Right now it supports FUSE mounts, but I'm thinking to make it work as a WebDAV server too. Also, I'm working on several demos, like SQLite compatibility, similar to https://github.com/lmatteis/torrent-net, or CSV analysis using Jupyter notebooks for huge datasets like https://ghtorrent.org/

Someone has built it already for bittorrent, maybe you can reuse some parts: https://github.com/lmatteis/torrent-net Even with an artificially slow connection it seems to be reactive enough. Actually fetching pages on-demand can only be better for the bandwidth, the real issue is going to be latency

A very interesting approach is sqltorrent (https://github.com/bittorrent/sqltorrent): the sqlite file is shared in a torrent, and all queries will touch a specific part of the file, which is downloaded on-demand. Also check https://github.com/lmatteis/torrent-net

I honestly think that the idea of letting people search inside a torrent before fully downloading it has been overlooked and might be the next big step in P2P file-sharing: https://github.com/lmatteis/torrent-net The idea is that you can actually query stuff inside a torrent; like actually perform an SQL query and only the pieces relevant to the query will be downloaded. This sort of allows for distributed querying if you think about it, since your query could be satisfied by many different peers that are seeding/leeching. Imagine a Web built this way. Where sites are served by people that visit them and are not just static sites, but fully queryable as you'd expect in the normal HTTP web. I wrote more on this here: https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4

This is awesome. If only we could share this huge dump of PDF files as a more structured format, perhaps using SQLite [1, 2], we could search through the torrents without having to wait for all of them to download beforehand. Although I guess the fact of having to "download them all beforehand" forces the data to be spread across various computers, hence increases availability of the data. One idea I had regarding this is perhaps structuring the contents of torrents as "append-only binary trees". So as new dumps are released every month, one can simply start downloading the torrent and has "search capabilities" for new data as well. 1. https://github.com/lmatteis/torrent-net 2. https://www.youtube.com/watch?v=EKttt8PYu5M&feature=youtu.be

Something like https://github.com/lmatteis/torrent-net ?

I think what he's more talking about is something more akin to a distributed search system than a blockchain - you could make the magnet links last forever in a blockchain, but that doesn't really make sense if no one holds the files anymore you're just keeping useless links around and storing them on every single node for no reason. A search DHT makes much more sense, look up keywords based on hash. Think the Kademlia network from the ED2K days. Maybe I'm being too academic by even suggesting a DHT be used, you could do it in simpler ways like flood search in the style of Gnutella - any distributed search means would work really. I believe there are a few attempts at this in the bittorrent world, like Tribler, which is probably the most practical implementation to date, there's a few others too, none of which look particularly mature yet

Next level, same idea but with SQLite: https://github.com/lmatteis/torrent-net