HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
BurntSushi
memchr
Rust2021–2026
BurntSushi / memchr

memchr

Optimized string search routines for Rust.

8Distinct people
24Mentions
0Explicit recommenders
1.6KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Optimized string search routines for Rust.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 8:59:02 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year24 dated mentions

Observed mentions: 2021-05-16 13:18:27 UTC to 2026-09-24 20:57:55 UTC.

0369122021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20212022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 11 mentions; 2 submissions, 9 comments20232024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 7 mentions; 1 submissions, 6 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC101
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC202
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC1129
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC211
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC716
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC101

What Hacker News says

11 sources
Right. I pointed it out because it isn't just about having portable SIMD that makes SIMD optimizations possible. Therefore, the lack of one in Rust doesn't have much explanatory power for why Rust's standard library doesn't contain SIMD. (It does have some.) It's good enough for things like memchr (well, kinda, NEON doesn't have `movemask`[1,2]), but not for things like Teddy that do multi-substring search. When you do want to write SIMD across platforms, it's not too hard to define your own bespoke portable API[3]. I'm basically just pointing out that a portable API is somewhat oversold, because it's not uncommon to need to abandon it, especially for string related ops that make creative use of ISA extensions. And additionally, that Rust unfortunately has other reasons for why std doesn't make as much use of SIMD as it probably should (t
Read the rest
In Ask HN: What's the fastest programming language with a large standard library?
Rust has had a stable SIMD vector API[1] for a long time. But, it's architecture specific. The portable API[2] isn't stable yet, but you probably can't use the portable API for some of the more exotic uses of SIMD anyway. Indeed, that's true in .NET's case too[3]. So .NET has to rely on "manually vectorized with intrisics for each individual platform" as well. Rust does all this SIMD too. It just isn't in the standard library. But the regex crate does it. Indeed, this is where .NET got its SIMD approach for multiple substring search from in the first place[4]. ;-) You're right that Rust's standard library is conservatively vectorized though[5]. The main thing blocking this isn't the lack of SIMD availability. It's more about how the standard library is internally structured, and the fact that things like substring search are not actually
Read the rest
defined in `std` directly, but rather, in `core`. There are plans to fix this[6]. With all that said, I wouldn't necessarily disagree with the broader point in this thread. .NET's regex engine performance is quite impressive[7]. (And I still have to merge the PR updating to .NET 8.) I'm really only replying here to correct more narrow points. [1]: https://doc.rust-lang.org/std/arch/index.html [2]: https://doc.rust-lang.org/std/simd/index.html [3]: https://github.com/dotnet/runtime/blob/72fae0073b35a404f03c3... [4]: https://github.com/dotnet/runtime/pull/88394#issuecomment-16... [5]: https://github.com/BurntSushi/memchr#why-is-the-standard-lib... [6]: https://github.com/rust-lang/rfcs/pull/3469 [7]: https://github.com/BurntSushi/rebar#summary
In Ask HN: What's the fastest programming language with a large standard library?
I agree with everything you said, but I don't see how it leads the OP's formulation being silly or wrong or not useful. Here's another example (of my own) where you can pick an upper bound for `n` and base your complexity analysis around it. In this case, we're trying to provide an API guarantee that a search takes O(m+n) time despite wanting to use an O(mn) algorithm in some subset of cases. We can still meet the O(m+n) bound by reasoning that the O(mn) algorithm is only used in a finite set of cases, and thus collapses to O(1) time. Therefore, the O(m+n) time bound is preserved. And this isn't a trick either. That really is the scaling behavior of the implementation. See: https://github.com/BurntSushi/memchr/blob/ce7b8e606410f6c81a... > If an algorithm is defined as being unscalable (fixed input), what sense does it make to describe tha
Read the rest
t it scales constantly with input size? I'll answer your question with another: in what cases does it make sense to describe the scaling behavior of an algorithm as O(1)?
In Rust-Cache
Copying my feedback from reddit[1], where I discussed it in the context of the `memchr` crate.[2] I took a quick look at your library implementation and have some notes: * It doesn't appear to query CPUID, so I imagine the only way it uses AVX2 on x86-64 is if the user compiles with that feature enabled explicitly. (Or uses something like [`x86-64-v3`](https://en.wikipedia.org/wiki/X86-64#Microarchitecture_level...).) The `memchr` crate doesn't need that. It will use AVX2 even if the program isn't compiled with AVX2 enabled so long as the current CPU supports it. * Your substring routines have multiplicative worst case (that is, `O(m * n)`) running time. The `memchr` crate only uses SIMD for substring search for smallish needles. Otherwise it flips over to Two-Way with a SIMD prefilter. You'll be fine for short needles, but things could g
Read the rest
o very very badly for longer needles. * It seems quite likely that your [confirmation step](https://github.com/ashvardanian/Stringzilla/blob/fab854dc4fd...) is going to absolutely kill performance for even semi-frequently occurring candidates. The [`memchr` crate utilizes information from the vector step to limit where and when it calls `memcmp`](https://github.com/BurntSushi/memchr/blob/46620054ff25b16d22...). Your code might do well in cases where matches are very rare. I took a quick peek at your benchmarks and don't see anything that obviously stresses this particular case. For substring search, the `memchr` crate uses a variant of the "[generic SIMD](http://0x80.pl/articles/simd-strfind.html#first-and-last)" algorithm. Basically, it takes two bytes from the needle, looks for positions where those occur and then attempts to check whether that position corresponds to a match. It looks like your technique uses the first 4 bytes. I suspect that might be overkill. (I did try using 3 bytes from the needle and found that it was a bit slower in some cases.) That is, two bytes is usually enough predictive power to lower the false positive rate enough. Of course, one can write pathological inputs that cause either one to do better than the other. (The `memchr` crat benchmark suite has a [collection of pathological inputs](https://github.com/BurntSushi/memchr/blob/46620054ff25b16d22...).) It would actually be possible to hook Stringzilla up to `memchr`'s benchmark suite if you were interested. :-) [1]: https://old.reddit.com/r/rust/comments/163ph8r/memchr_26_now... [2]: https://github.com/BurntSushi/memchr
In Stringzilla: Fastest string sort, search, split, and shuffle using SIMD
I actually have an M2 mac mini in the mail from Apple for exactly this purpose. My time horizon is very long. It takes me a long time to do things these days. It has never been true that I don't want to support it. Merely that it is difficult to verify and test if I can't do it myself. There is also the problem that the port from x86 to arm is not straight-forward, due to both my own ignorance and what I believe are important missing vector operations such as movemask. This is discussed a bit more here (including the bit about movemask): https://github.com/BurntSushi/memchr/issues/76
In Regex engine internals as a library
ripgrep author here. Very interesting! I tried reproducing it, but unfortunately: $ sneller -j -S "SELECT COUNT(Line) FROM 'en.json' WHERE Line ~ 'Sherlock [A-Z]\\w+'" expr.Unescape: unexpected backslash escape of 'w' (0x77) $ sneller -j -S "SELECT COUNT(Line) FROM 'en.json' WHERE Line ~ 'Sherlock [A-Z][a-z]+'" CPU doesn't support AVX-512 And that is the primary reason why ripgrep doesn't bother with AVX-512. Not because of some lack of skill as this blog suggests: > Additionally, ripgrep uses AVX2 and does not take advantage of AVX-512 instruction sets, but this can be forgiven given the specialized skills required for handcrafting for SkylakeX and Icelake/Zen4 processors. Namely, I tried running sneller on my CPU, which is a pretty recent i9-12900K, and not even it supports AVX-512. That's because Intel has been droppi
Read the rest
ng support for AVX-512 from its more recent consumer grade CPUs. ripgrep is running far more frequently on consumer grade CPUs, so supporting AVX-512 is probably not particularly advantageous. At least, it's not obvious to me that it's worth doing. And certainly, the skill argument isn't entirely wrong. I'd have to invest developer time to make it work. I think there are two other things worth highlighting from this blog. First is that sneller seems to do quite well with compressed data. This is definitely not ripgrep's strong suit. When you use ripgrep's -z/--search-zip flag, all it's doing is shelling out to your gzip/xz/whatever executable to do the decompression work, which is then streamed into ripgrep for searching. So if your search speed tanks when using -z/--search-zip, it's likely because your decompression tools are slow, not because of ripgrep. But it's a fair comparison from sneller's perspective, because it seems to integrate the two. Second is the issue of multi-threaded search. In ripgrep, the fundamental unit of work is "search a file." ripgrep has no support for more granular parallelism. That is, if you give it one file, it's limited to doing a single threaded search. ripgrep could do more granular parallelism, but it hasn't been obviously worth it to me. If most searches are on a directory tree, then parallelizing at the level of each file is almost certainly good enough. Making ripgrep's parallelism more fine grained is a fair bit of work too, and there would be a lot of fiddly stuff to get right. If I could run sneller easily, I'd probably try to see how it does in a more varied workload than what is presented in this blog. :-) And finally some corrections: > However, when using a single thread, ripgrep appears to be slightly faster. Not just slightly faster, over 2x faster! The single threaded results for Regex2 and Regex3 for Sneller are quite nice! I'd be interested in hearing more about what you're doing in the Regex2 case, since Sneller and ripgrep are about on par with the Regex3 case. Maybe a fail fast optimization? > The reason for this is that ripgrep uses the Boyer-Moore string search algorithm, which is a pattern matching algorithm that is often used for searching for substrings within larger strings. It is particularly efficient when the pattern being searched for is relatively long and the alphabet of characters being searched over is relatively small. Sneller does not use this substring search algorithm and as a result is slower than ripgrep with substrings. However, when long substrings are not present, Sneller outperforms ripgrep. ripgrep has never used Boyer-Moore. (Okay, some years ago, ripgrep could use Boyer-Moore in certain niche cases. But that hasn't been the case for a while and it was never the thing most commonly used). What ripgrep uses today is succinctly described here: https://github.com/BurntSushi/memchr#algorithms-used (But it has always eschewed algorithms like Boyer-Moore in favor of more heuristic-y approaches based on a background frequency distribution of bytes.) I think I would also contest the claim that "long substrings" are the key here. ripgrep is plenty fast with short substrings too. You're correct that if you have no literals then ripgrep will get slower because it has to fall back to the regex engine. But I'd like to see more robust benchmarks there. Your Regex2 and Regex3 benchmarks raise more questions than it answers. :-) > Although the resulting .dot and .svg files may be somewhat clunky, we can still observe from the graph that the number of nodes and edges are small enough to use the branchless IceLake implementation. In this particular case, we only need 8 bits to encode the number of nodes and the number of distinct edges, enabling the tool to use (what we call) the 8-bit DFA implementation. For more details on how this works, see our post on regex implementations. So this is talking about the DFA graph for the regex `Sherlock [A-Z]\w+`. It's important to point out that, in ripgrep, `\w` is Unicode aware by default. Which makes it absolutely enormous. So I think the state graph you linked is probably only for the ASCII version of that regex. Indeed, reading your regex blog[1], it perhaps looks like a lot of the tricks you use won't work for Unicode, because Unicode tends to blow up finite automata. If I could run Sneller, I'd probably try to poke it to see what its Unicode support looks like. From a quick glance of the source code, it also looks like you build full DFAs. So I would also try to poke it to see what happens when handed a particularly a not-so-small regex. (Building a DFA can take quite some time.) Ah okay, I see, you put a max limit on the DFA: https://github.com/SnellerInc/sneller/blob/bb5adec564bf9869d... Overall this is a very cool project! Also, is the regex engine available as a standalone library? It would be interesting to benchmark it on its own. Although, if it only works when AVX-512 is available, then that is pretty limiting unfortunately. [1]: https://sneller.io/blog/accelerating-regex-using-avx-512/
In Sneller Regex vs Ripgrep
Indeed. This is how ripgrep works. It's compiled for just plain `x86_64`, but it looks for whether things like AVX2 are enabled. And if so, uses vector algorithms for substring and multi-substring search. The nice thing about dealing with strings is that the "coarse" requirement is already somewhat natural to the domain. But, this functionality is absolutely critical. It doesn't even have to be automatic. Just the ability to compile functions with certain ISA extensions enabled, and then only call them when the requisite CPU features are enabled is enough. In a nutshell: https://github.com/BurntSushi/memchr/blob/8037d11b4357b0f07b...
In SIMD with Zig

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.