HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
wilsonzlin
edgesearch
Rust2021–2022
wilsonzlin / edgesearch

edgesearch

Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps

Web DevelopmentSystems & InfrastructureDevOps & CloudSearch & Information Retrieval
Explore on GitHubRead Hacker News discussions
4Distinct people
7Mentions
0Explicit recommenders
572GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month7 dated mentions

Observed mentions: 2021-01-06 22:05:48 UTC to 2022-04-04 16:43:32 UTC.

0122021-01-01 00:00:00 UTC to 2021-02-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 commentsJan 20212021-02-01 00:00:00 UTC to 2021-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-03-01 00:00:00 UTC to 2021-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-04-01 00:00:00 UTC to 2021-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-05-01 00:00:00 UTC to 2021-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-06-01 00:00:00 UTC to 2021-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-07-01 00:00:00 UTC to 2021-08-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2021-08-01 00:00:00 UTC to 2021-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsAug 20212021-09-01 00:00:00 UTC to 2021-10-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2021-10-01 00:00:00 UTC to 2021-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-11-01 00:00:00 UTC to 2021-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-12-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-01-01 00:00:00 UTC to 2022-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-02-01 00:00:00 UTC to 2022-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-03-01 00:00:00 UTC to 2022-04-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2022-04-01 00:00:00 UTC to 2022-05-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsApr 2022

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Jan 20212021-01-01 00:00:00 UTC2021-02-01 00:00:00 UTC211
Feb 20212021-02-01 00:00:00 UTC2021-03-01 00:00:00 UTC000
Mar 20212021-03-01 00:00:00 UTC2021-04-01 00:00:00 UTC000
Apr 20212021-04-01 00:00:00 UTC2021-05-01 00:00:00 UTC000
May 20212021-05-01 00:00:00 UTC2021-06-01 00:00:00 UTC000
Jun 20212021-06-01 00:00:00 UTC2021-07-01 00:00:00 UTC000
Jul 20212021-07-01 00:00:00 UTC2021-08-01 00:00:00 UTC202
Aug 20212021-08-01 00:00:00 UTC2021-09-01 00:00:00 UTC000
Sep 20212021-09-01 00:00:00 UTC2021-10-01 00:00:00 UTC101
Oct 20212021-10-01 00:00:00 UTC2021-11-01 00:00:00 UTC000
Nov 20212021-11-01 00:00:00 UTC2021-12-01 00:00:00 UTC000
Dec 20212021-12-01 00:00:00 UTC2022-01-01 00:00:00 UTC000
Jan 20222022-01-01 00:00:00 UTC2022-02-01 00:00:00 UTC000
Feb 20222022-02-01 00:00:00 UTC2022-03-01 00:00:00 UTC000
Mar 20222022-03-01 00:00:00 UTC2022-04-01 00:00:00 UTC101
Apr 20222022-04-01 00:00:00 UTC2022-05-01 00:00:00 UTC101

What Hacker News says

7 sources
westurner·Apr 4, 2022
From https://westurner.github.io/hnlog/#comment-30603322 re https://en.wikipedia-on-ipfs.org/wiki/Database_index : https://github.com/wilsonzlin/edgesearch : > * Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps * > "Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers" WASM or [C++] to WASM? TIL about Roaring Bitmaps: /?q=roaring+bitmap https://medium.com/@amit.desai03/roaring-bitmaps-fast-data-s...
In Lifetime Annotations for C++
westurner·Mar 8, 2022
https://en.wikipedia.org/wiki/Database_index https://en.wikipedia-on-ipfs.org/wiki/Database_index From "Hosting SQLite Databases on GitHub Pages" https://news.ycombinator.com/item?id=28021766 re: edgesearch, HTTP/3 QUIC UDP, : > Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps https://github.com/wilsonzlin/edgesearch >> How it works: Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers
In How does database indexing work? (2008)
ignoramous·Sep 27, 2021
Neat! > Incidentally searching around for prior art I found this blog post https://www.morling.dev/blog/how-i-built-a-serverless-search... about building something similar using lucene, but without storing the content and only on a single lambda. See also: Sqlite FTS (no server): https://news.ycombinator.com/item?id=27016630 EdgeSearch (Workers): https://github.com/wilsonzlin/edgesearch EdgeSql (Workers): https://github.com/lspgn/edge-sql InfiniCache (Lambda): https://news.ycombinator.com/item?id=25788893 quickwit (Lambda): https://news.ycombinator.com/item?id=27074481 > I had been working on a bloom filter based index based on the ideas of bitfunnel which was developed by Bob Goodwin, Michael Hopcroft, Dan Luu, Alex Clemmer, Mihaela Curmei, Sameh Elnikety and Yuxiong He and used in Microsoft Bing https://danluu.com/bitfunnel-sigir.pdf. R
Read the rest
eminds me of wavelet trees: https://alexbowe.com/succinct-debruijn-graphs/ https://alexbowe.com/wavelet-trees/
In Abusing AWS Lambda to make an Aussie search engine
westurner·Jul 31, 2021
> Methods for remotely accessing/paging data in from a client when a complete download of the dataset is unnecessary: > - Query e.g. parquet on e.g. GitHub with DuckDB: duckdb/test_parquet_remote.test https://github.com/duckdb/duckdb/blob/6c7c9805fdf1604039ebed... > - Query sqlite on e.g. GitHub with SQLite: [Hosting SQLite databases on Github Pages - (or any static file hoster) - phiresky's blog](...) >> The above query should do 10-20 GET requests, fetching a total of 130 - 270KiB, depending on if you ran the above demos as well. Note that it only has to do 20 requests and not 270 (as would be expected when fetching 270 KiB with 1 KiB at a time). That’s because I implemented a pre-fetching system that tries to detect access patterns through three separate virtual read heads and exponentially increases the request size for sequential re
Read the rest
ads. This means that index scans or table scans reading more than a few KiB of data will only cause a number of requests that is logarithmic in the total byte length of the scan. You can see the effect of this by looking at the “Access pattern” column in the page read log above. > - bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent >> Sqltorrent is a custom VFS for sqlite which allows applications to query an sqlite database contained within a torrent. Queries can be processed immediately after the database has been opened, even though the database file is still being downloaded. Pieces of the file which are required to complete a query are prioritized so that queries complete reasonably quickly even if only a small fraction of the whole database has been downloaded. >> […] Creating torrents: Sqltorrent currently only supports torrents containing a single sqlite database file. For efficiency the piece size of the torrent should be kept fairly small, around 32KB. It is also recommended to set the page size equal to the piece size when creating the sqlite database Would BitTorrent be faster over HTTP/3 (UDP) or is that already a thing for web seeding? > - https://web.dev/file-system-access/ > The File System Access API: simplifying access to local files: The File System Access API allows web apps to read or save changes directly to files and folders on the user’s device Hadn't seen wilsonzlin/edgesearch, thx: > Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps https://github.com/wilsonzlin/edgesearch >> How it works: Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers
In Hosting SQLite Databases on GitHub Pages
ignoramous·Jul 31, 2021
> If I may suggest one thing... instead of range requests on a single huge file how about splitting the file in 1-page fragments in separate files and fetching them individually? Edgesearch does that though with Cloudflare Workers mediating searches: https://github.com/wilsonzlin/edgesearch Uses roaring-bitmaps to index, but could also use Stavros' trick (bloom/cuckoo-filters) to further gate false-positives on-the-client: https://news.ycombinator.com/item?id=23473365
In Hosting SQLite Databases on GitHub Pages
tmikaeld·Jan 20, 2021
I've also tried full-text-search in worker by pre-indexing the content, works very fast even with a JS engine - less than 5ms to make a search in 5MB of text. It runs out of CPU-time at 6MB of index though. There's someone that made a WASM for full-text search too, it's definitely faster and can handle quite a lot more text. https://github.com/wilsonzlin/edgesearch
In Serverless SQLite

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.