imdb-rename

What a shit take. The article itself is perhaps a nice light overview of 101-ish level concepts, although knowing how and when to apply them in a real engineering context is not something I would consider 101 level. And certainly, building something that is actually at the scale of GitHub Search is nowhere near 101 level. This is what a 101-level inverted index implementation looks like: https://github.com/BurntSushi/imdb-rename In other words, absolutely nothing like what GitHub built. Nowhere close.

No you don't. I've written multiple programs that load things instantly off the file system via memory maps. See the fst crate[1], for example, which is designed to work with memory maps. imdb-rename[2] is a program I wrote that builds a simple IR index on your file system that can then instantly search it by virtue of memory maps. Rust "works badly with memory mapped files" doesn't mean, "Rust can't use memory mapped files." It means, "it is difficult to reconcile Rust's safety story with memory maps." ripgrep for example uses memory maps because they are faster sometimes, and its safety contract[3] is a bit strained. But it works. [1] - https://github.com/BurntSushi/fst/ [2] - https://github.com/BurntSushi/imdb-rename [3] - https://docs.rs/grep-searcher/0.1.7/grep_searcher/struct.Mma...

> How else would you lazy-load a database of (say) 32GB into memory, almost instantly? That's what the fst crate[1] does. It's likely working at a lower level of abstraction than you intend. But the point is that it works, is portable and doesn't require any cooperation from the OS other than the ability to memory map files. My imdb-rename tool[2] uses this technique to build an on-disk database for instantaneous searching. And then there is the regex-automata crate[3] that permits deserializing a regex instantaneously from any kind of slice of bytes.[4] I think you should maybe provide some examples of what you're suggesting to make it more concrete. [1] - https://crates.io/crates/fst [2] - https://github.com/BurntSushi/imdb-rename [3] - https://crates.io/crates/regex-automata [4] - https://docs.rs/regex-automata/0.1.9/regex_automata/#ex

If you liked goim, then you'll definitely want to check out its replacement: https://github.com/BurntSushi/imdb-rename/ It's even faster and comes with its own built in evaluation. There's also no more reliance on an external database.