bloomsearch

Reminds me of @danthegoodman's project: bloomsearch [1] [1] https://github.com/danthegoodman1/bloomsearch

The BloomSearchEngine takes a TokenizerFunc so you can determine how JSON values are tokenized (that's why each path always returns an array of strings). The default tokenizer is a a whitespace one: https://github.com/danthegoodman1/bloomsearch/blob/148a79967... So {"name": "John Smith"} is tokenized to [{Path: "name", Values: ["john", "smith"]}], and the bloom filters will store: - field: "name" - token: "john" - token: "smith" - fieldtoken: "name:john" - fieldtoken: "name:smith" The same tokenizer must be used at query time too. Fuzzy searches and sub-word searches could be supported with custom tokenizers (eg trigrams, stemming), but it's more generally targeting the "I know some exact subset of the record, I need all that have this exactly" searches

Hey HN! I got nerd-sniped by Bloom Filters this weekend, specifically for searching datasets with high "cardinality" (number of unique items). They're an _amazing_ data structure that, at a fixed size, tracks potential set membership. That means unlike normal b-tree indexes, they don't grow with the number of unique items in the dataset. This makes them great for "needle in a haystack" search (logs, document) as implementations like VictoriaMetrics and Bing's BitFunnel show. I've used them in the past, but they've never been center-stage in my projects. I wanted high cardinality keyword search for ANOTHER project... and, well, down the yak-shaving rabbit hole we go! BloomSearch brings this into an extensible Go package: - Very memory efficient via bloom filters and streaming row scans - DataStore and MetaStore interfaces for any backend (