datasette-lite

ASM with pyodide. https://github.com/simonw/datasette-lite pygwalker: https://github.com/Kanaries/pygwalker .. https://news.ycombinator.com/item?id=35895899 How do you record manual interactions with ui controls and spreadsheet grids to code for reproducibility? > "Generate code from GUI interactions; State restoration & Undo" https://github.com/Kanaries/pygwalker/issues/90 > The Scientific Method is testing, so testing (tests, assertions, fixtures) should be core to any scientific workflow system. ipytest has a %%ipytest cell magic to run functions that start with test_ and subclasses of unittest.TestCase with the pytest test runner. https://github.com/chmp/ipytest How can test functions with assertions be written with Probly?

html#cross-d... : > SQLite has the ability to run queries that join across multiple databases. Up to ten databases can be attached to a single SQLite connection and queried together.

vfs was a bit more of a POC, and later some possibly better solutions came out, but I haven't really went that rabbit hole to figure out which one would be best. Edit: looking a little more into datasette-lite, it seems like one of the nice benefits of sql.js-httpvfs is that it doesn't download the whole SQLite database in order to query it. This makes it possible have a 2GB database but still read it in chunks, skipping around efficiently until you find your data.

s sqlite in notebooks in the browser with `import sqlite3` with the python kernel and also with a sqlite kernel: https://jupyter.org/try-jupyter/lab/ jupyterlite/xeus-sqlite-kernel: https://github.com/jupyterlite/xeus-sqlite-kernel (edit) xeus-sqlite-kernel > "Loading SQLite databases from a remote URL" https://github.com/jupyterlite/xeus-sqlite-kernel/issues/6#i... %FETCH <url> <filename> https://github.com/jupyter-xeus/xeus-sqlite/blob/ce5a598bdab... xlite.cpp > void fetch(const std::string url, const std::string filename) https://github.com/jupyter-xeus/xeus-sqlite/blob/main/src/xl...

holarlyArticle by optional :author, and now there are W3C DIDs Decentralized Identifiers for signing, identifying, and searching of unique :Thing and skos:Concept that can be generated offline and optionally registered centrally, or centrally generated and assigned like DOIs but they're signing keys. Given uncertainty about time intervals, plot concepts over time with charts for showing graph growth over time. Maybe philosophy skos:Concept intervals (and relations, links) from human annotations thereof and/or from LLM parsing and search snippets of Wikipedia, dbpedia RDF, wikidata RDF, and ranked Stanford Encyclopedia of Philosophy terminological occurrence frequency.

live-dem... : datasette publish vercel fixtures.db [...] The `datasette publish` command supports Google Cloud Run, Heroku, Vercel, Fly, [Full Container or Serverless] https://docs.datasette.io/en/stable/publish.html datasette-lite: https://github.com/simonw/datasette-lite : > You can use this tool to open any SQLite database file that is hosted online and served with a `access-control-allow-origin: ` CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using `datasette publish`.* > [...] You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you > To load a Parquet file, pass a URL to `?parquet=` > [...] https://lite.datasette.io/?parquet=https://github.com/Terada...* There are various *-to-sqlite utilities that load data into a SQLite database for use with e.g. datasette. E.g. Pandas with `dtype_backend='arrow'` saves to Parquet. datasette plugins are written in Python and/or JS w/ pluggy: https://docs.datasette.io/en/stable/plugins.html https://datasette.io/plugins datasette-scraper scrapes sitemaps.xml and crawls though it could surely be repurposed to instead scrape a list of code.json URLs within the datasette process, which is powered by asyncio and the asynchronous uvicorn ASGI HTTP web server. datasette-scraper/#architecture: https://github.com/cldellow/datasette-scraper/#architecture (TIL datasette-scraper parses HTML with selectolax; and Selectolax with Modest or Lexbor is ~25x faster at HTML parsing than BeautifulSoup in the selectolax benchmark: https://github.com/rushter/selectolax#simple-benchmark ) (Apache Nutch is a Java-based web crawler which supports e.g. CommonCrawl (which backs various foundational LLMs)) https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_bu... . But extruct extracts more types of metadata and data than Nutch AFAIU: https://github.com/scrapinghub/extruct ) datasette-graphql adds a GraphQL HTTP API to a SQLite database: https://datasette.io/plugins/datasette-graphql plugins?q=sqlite: https://datasette.io/plugins?q=sqlite datasette-sqlite-fts4: https://datasette.io/plugins/datasette-sqlite-fts4 ; Full-Text Search with SQLite datasette-ripgrep: "deploy a regular expression search engine for your source code": https://github.com/simonw/datasette-ripgrep Seeing as there's already a JSONLD @context (schema) for code.json, CSVW as JSONLD and/or YAMLLD would be an easy way merge Linked Data graphs of tabular data: https://github.com/semantalytics/awesome-semantic-web#csvw A GitHub Action would run regularly, fetch each code.json, save each to a git repo, and then upsert each into a SQLite database to be published with e.g. datasette or datasette-lite.

olt can be set up as a replica of your existing MySQL or MariaDB database using standard MySQL binlog replication. Every write becomes a Dolt commit. This is a great way to get the version control benefits of Dolt and keep an existing MySQL or MariaDB database. #. github/gh-ost: https://github.com/github/gh-ost : > Instead, gh-ost uses the binary log stream to capture table changes, and asynchronously applies them onto the ghost table. gh-ost takes upon itself some tasks that other tools leave for the database to perform. As result, gh-ost has greater control over the migration process; can truly suspend it; can truly decouple the migration's write load from the master's workload. #. vlcn-io/cr-sqlite: https://github.com/vlcn-io/cr-sqlite : > Convergent, Replicated SQLite. Multi-writer and CRDT support for SQLite > CR-SQLite is a run-time loadable extension for SQLite and libSQL. It allows merging different SQLite databases together that have taken independent writes. > In other words, you can write to your SQLite database while offline. I can write to mine while offline. We can then both come online and merge our databases together, without conflict. > In technical terms: cr-sqlite adds multi-master replication and partition tolerance to SQLite via conflict free replicated data types (CRDTs) and/or causally ordered event logs. yjs also does CRDTs (Jupyter RTC,) #. pganalyze/libpg_query: https://github.com/pganalyze/libpg_query : > C library for accessing the PostgreSQL parser outside of the server environment #. Ibis + Substrait [ + DuckDB ] https://ibis-project.org/blog/ibis_substrait_to_duckdb/ : > ibis strives to provide a consistent interface for interacting with a multitude of different analytical execution engines, most of which (but not all) speak some dialect of SQL. > Today, Ibis accomplishes this with a lot of help from `sqlalchemy` and `sqlglot` to handle differences in dialect, or we interact directly with available Python bindings (for instance with the pandas, datafusion, and polars backends). > [...] `Substrait` is a new cross-language serialization format for communicating (among other things) query plans. It's still in its early days, but there is already nascent support for Substrait in Apache Arrow, DuckDB, and Velox. #. ibis-project/ibis-substrait: https://github.com/ibis-project/ibis-substrait #. tobymao/sqlglot: https://github.com/tobymao/sqlglot : > SQLGlot is a no-dependency SQL parser, transpiler, optimizer, and engine. It can be used to format SQL or translate between 19 different dialects like DuckDB, Presto, Spark, Snowflake, and BigQuery. It aims to read a wide variety of SQL inputs and output syntactically and semantically correct SQL in the targeted dialects. > It is a very comprehensive generic SQL parser with a robust test suite. It is also quite performant, while being written purely in Python. > You can easily customize the parser, analyze queries, traverse expression trees, and programmatically build SQL. > Syntax errors are highlighted and dialect incompatibilities can warn or raise depending on configurations. However, it should be noted that SQL validation is not SQLGlot’s goal, so some syntax errors may go unnoticed. #. benbjohnson/postlite: https://github.com/benbjohnson/postlite : > postlite is a network proxy to allow access to remote SQLite databases over the Postgres wire protocol. This allows GUI tools to be used on remote SQLite databases which can make administration easier. > The proxy works by translating Postgres frontend wire messages into SQLite transactions and converting results back into Postgres response wire messages. Many Postgres clients also inspect the pg_catalog to determine system information so Postlite mirrors this catalog by using an attached in-memory database with virtual tables. The proxy also performs minor rewriting on these system queries to convert them to usable SQLite syntax. > Note: This software is in alpha. Please report bugs. Postlite doesn't alter your database unless you issue INSERT, UPDATE, DELETE commands so it's probably safe. If anything, the Postlite process may die but it shouldn't affect your database. #. > "Hosting SQLite Databases on GitHub Pages" (2021) re: sql.js-httpvfs, DuckDB https://news.ycombinator.com/item?id=28021766 #. >> - bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent >> Sqltorrent is a custom VFS for sqlite which allows applications to query an sqlite database contained within a torrent. Queries can be processed immediately after the database has been opened, even though the database file is still being downloaded. Pieces of the file which are required to complete a query are prioritized so that queries complete reasonably quickly even if only a small fraction of the whole database has been downloaded. #. simonw/datasette-lite: https://github.com/simonw/datasette-lite datasette, *-to-sqlite, dogsheep "Loading SQLite databases" [w/ datasette] https://github.com/simonw/datasette-lite#loading-sqlite-data... #. awesome-db-tools: https://github.com/mgramin/awesome-db-tools Lots of neat SQLite/vtable/pg/replication things

s have been ported from pyodide. > While we already have a lot of packages built, this is still a big work in progress.

b.com/simonw/datasette-lite/pull/49. It uses a hacked up chunk'd lazyFile implementation from emscripten and others to grab pages from Cloudflare R2. Here's a test/demo with california's unclaimed property records (https://www.sco.ca.gov/upd_download_property_records.html) of a 28GB searching up that guy who owns Twitter: https://datasette-lite-lab.mindflakes.com/index.html?url=htt...

html?url=htt... Still, not bad for a $0.42/mo hostable cached CDN'd read-only database. It's on Cloudflare R2, so there's no BW costs.