HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
BurntSushi
memchr
Rust2021–2026
BurntSushi / memchr

memchr

Optimized string search routines for Rust.

8Distinct people
24Mentions
0Explicit recommenders
1.6KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Optimized string search routines for Rust.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 8:59:02 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year24 dated mentions

Observed mentions: 2021-05-16 13:18:27 UTC to 2026-09-24 20:57:55 UTC.

0369122021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20212022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 11 mentions; 2 submissions, 9 comments20232024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 7 mentions; 1 submissions, 6 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC101
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC202
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC1129
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC211
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC716
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC101

What Hacker News says

7 sources
It's not quite that dire. The `memchr` crate uses abstraction to limit the code duplication: https://github.com/BurntSushi/memchr/blob/1230fc5c638a4d922f... That is, the `memchr` crate has a `Vector` trait that is generic over the vector type. Which is essentially your `DataStructure<S>` where `S` is the ISA. (Using the vector type isn't load bearing. I could do the ISA. But in `memchr`'s case, the vector type implies the ISA.) It relies on `#[inline(always)]` to work. But you can write the algorithm generically once: https://github.com/BurntSushi/memchr/blob/1230fc5c638a4d922f... And the entry point into a specific instantiation of the generic algorithm is where you apply `#[target_feature(enable = "foo")]`: https://github.com/BurntSushi/memchr/blob/1230fc5c638a4d922f...
In The state of SIMD in Rust in 2025
Nope. ripgrep uses the `memchr` crate for substring search, and in my benchmarks it's generally faster than stringzilla: $ rebar cmp results.csv --intersection -f huge benchmark rust/memchr/memmem/prebuilt stringzilla/memmem/oneshot --------- --------------------------- -------------------------- memmem/pathological/md5-huge-no-hash 47.4 GB/s (1.00x) 38.1 GB/s (1.25x) memmem/pathological/md5-huge-last-hash 40.3 GB/s (1.00x) 23.4 GB/s (1.72x) memmem/pathological/rare-repeated-huge-tricky 40.4 GB/s (1.04x) 42.0 GB/s (1.00x) memmem/pathological/rare-repeated-huge-match 1977.7 MB/s (1.00x) 563.3 MB/s (3.51x) memmem/subtitles/common/huge-en-that 35.9 G
Read the rest
B/s (1.00x) 25.3 GB/s (1.42x) memmem/subtitles/common/huge-en-you 15.9 GB/s (1.00x) 9.5 GB/s (1.67x) memmem/subtitles/common/huge-en-one-space 1376.4 MB/s (1.00x) 1364.0 MB/s (1.01x) memmem/subtitles/common/huge-ru-that 29.0 GB/s (1.00x) 15.5 GB/s (1.87x) memmem/subtitles/common/huge-ru-not 16.0 GB/s (1.00x) 3.5 GB/s (4.53x) memmem/subtitles/common/huge-ru-one-space 2.6 GB/s (1.00x) 2.4 GB/s (1.08x) memmem/subtitles/common/huge-zh-that 31.2 GB/s (1.00x) 23.8 GB/s (1.31x) memmem/subtitles/common/huge-zh-do-not 19.4 GB/s (1.00x) 12.1 GB/s (1.59x) memmem/subtitles/common/huge-zh-one-space 5.3 GB/s (1.05x) 5.6 GB/s (1.00x) memmem/subtitles/never/huge-en-john-watson 41.2 GB/s (1.00x) 31.2 GB/s (1.32x) memmem/subtitles/never/huge-en-all-common-bytes 47.9 GB/s (1.00x) 37.5 GB/s (1.28x) memmem/subtitles/never/huge-en-some-rare-bytes 43.4 GB/s (1.00x) 42.7 GB/s (1.02x) memmem/subtitles/never/huge-en-two-space 42.2 GB/s (1.00x) 30.7 GB/s (1.37x) memmem/subtitles/never/huge-ru-john-watson 42.2 GB/s (1.00x) 42.1 GB/s (1.00x) memmem/subtitles/never/huge-zh-john-watson 47.6 GB/s (1.00x) 34.0 GB/s (1.40x) memmem/subtitles/rare/huge-en-sherlock-holmes 40.8 GB/s (1.05x) 42.9 GB/s (1.00x) memmem/subtitles/rare/huge-en-sherlock 36.7 GB/s (1.16x) 42.5 GB/s (1.00x) memmem/subtitles/rare/huge-en-medium-needle 47.7 GB/s (1.00x) 31.3 GB/s (1.52x) memmem/subtitles/rare/huge-en-long-needle 44.5 GB/s (1.00x) 32.0 GB/s (1.39x) memmem/subtitles/rare/huge-en-huge-needle 45.7 GB/s (1.00x) 33.4 GB/s (1.37x) memmem/subtitles/rare/huge-ru-sherlock-holmes 42.1 GB/s (1.00x) 42.2 GB/s (1.00x) memmem/subtitles/rare/huge-ru-sherlock 42.3 GB/s (1.01x) 42.9 GB/s (1.00x) memmem/subtitles/rare/huge-zh-sherlock-holmes 46.7 GB/s (1.00x) 33.1 GB/s (1.41x) memmem/subtitles/rare/huge-zh-sherlock 47.4 GB/s (1.00x) 42.8 GB/s (1.11x) But I would say they are overall pretty competitive. If you want to run the benchmarks yourself, you can. First, get rebar[1]. Then, from the root of the `memchr` repository[2]: $ rebar build -e 'rust/memchr/memmem/prebuilt' -e 'stringzilla/memmem/oneshot' stringzilla/memmem/oneshot: running: cd "benchmarks/./engines/stringzilla" && "cargo" "build" "--release" stringzilla/memmem/oneshot: build complete for version 3.12.3 rust/memchr/memmem/prebuilt: running: cd "benchmarks/./engines/rust-memchr" && "cargo" "build" "--release" rust/memchr/memmem/prebuilt: build complete for version 2.7.4 $ rebar measure -e 'rust/memchr/memmem/prebuilt' -e 'stringzilla/memmem/oneshot' | tee results.csv $ rebar rank results.csv Engine Version Geometric mean of speed ratios Benchmark count ------ ------- ------------------------------ --------------- rust/memchr/memmem/prebuilt 2.7.4 1.14 57 stringzilla/memmem/oneshot 3.12.3 1.43 54 $ rebar cmp results.csv --intersection -f never/huge benchmark rust/memchr/memmem/prebuilt stringzilla/memmem/oneshot --------- --------------------------- -------------------------- memmem/subtitles/never/huge-en-john-watson 41.2 GB/s (1.00x) 31.2 GB/s (1.32x) memmem/subtitles/never/huge-en-all-common-bytes 47.9 GB/s (1.00x) 37.5 GB/s (1.28x) memmem/subtitles/never/huge-en-some-rare-bytes 43.4 GB/s (1.00x) 42.7 GB/s (1.02x) memmem/subtitles/never/huge-en-two-space 42.2 GB/s (1.00x) 30.7 GB/s (1.37x) memmem/subtitles/never/huge-ru-john-watson 42.2 GB/s (1.00x) 42.1 GB/s (1.00x) memmem/subtitles/never/huge-zh-john-watson 47.6 GB/s (1.00x) 34.0 GB/s (1.40x) See also: https://github.com/BurntSushi/memchr/discussions/159 [1]: https://github.com/BurntSushi/rebar [2]: https://github.com/BurntSushi/memchr
In Processing Strings 109x Faster Than Nvidia on H100
The "AVX2 (generic)" approach is roughly what ripgrep uses (via Rust's `regex` crate) to accelerate most searches. Even something like `\w+\s+Sherlock\s+\w+` will benefit since ripgrep will pluck `Sherlock` out of the regex and search that. The actual implementation is here: https://github.com/BurntSushi/memchr?tab=readme-ov-file#algo... The main difference with the algorithm presented here is that instead of always using the first and last bytes of the needle, a heuristic is used to try to pick two bytes that occur less frequently according to a background frequency distribution. It ends up being quite a bit faster than just plain Two-Way or even GNU libc's implementation of `memmem`. From the root of the `memchr` repository: $ rebar rank benchmarks/record/x86_64/2023-12-29.csv -e '^rust/memchr/memmem/(oneshot|prebuilt|twoway)' -e '^
Read the rest
libc/memmem/oneshot' Engine Version Geometric mean of speed ratios Benchmark count ------ ------- ------------------------------ --------------- rust/memchr/memmem/prebuilt 2.7.1 1.07 57 rust/memchr/memmem/oneshot 2.7.1 1.39 54 libc/memmem/oneshot unknown 3.15 54 rust/memchr/memmem/twoway 2.7.1 5.26 54 The "prebuilt" benchmarks also demonstrate the API deficiencies of routines like `memmem`. Because of their API, they need to rebuild the state necessary to execute a search for the needle given. But it is often the case that the needle is invariant across repeated calls with different haystacks. So you end up doing that repeated work for no gain.
In SIMD-friendly algorithms for substring searching (2016)
That's probably because pcmpestri is trash for substring search. There is a good reason why ripgrep doesn't use it. :-) I looked for an authoritative search for why pcmpestri is trash, and I couldn't find anything I was happy linking to other than Agner Fog's instruction tables: https://www.agner.org/optimize/instruction_tables.pdf You can see that the throughput and latency for pcmpestri is just awful. And yes, not having any code to print the matching lines means that the only code path in krep is just counting things. If that's all your tool is doing, you can totally beat ripgrep or any other tool that is more applicable to generalized use cases. It's why the `memchr` crate (what ripgrep uses for single substring search) has a specialized routine for counting occurrences of bytes (which ripgrep uses for line counting): https://github.c
Read the rest
om/BurntSushi/memchr/blob/746182171d2e886006... Because it's faster to do that than it is to reuse the generalized `memchr` API for finding the location of matching bytes. And counting matches in a multi-threaded context is way easier than actual managing the printing of matches in the same order that you get them. krep isn't big. You can skim its source code in a few minutes and get a good idea of how it works.
In Show HN: Krep a High-Performance String Search Utility Written in C
This wouldn't build for me, so I had to apply the patch suggested by a sibling comment. Once I got it building, my first benchmark attempt shows it as being slower: $ curl -LO 'https://burntsushi.net/stuff/subtitles2016-sample.en.gz' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 265M 100 265M 0 0 48.6M 0 0:00:05 0:00:05 --:--:-- 49.9M $ gzip -d subtitles2016-sample.en.gz $ hyperfine --ignore-failure "rg -c 'ZQZQZQZQ' subtitles2016-sample.en" "krep -c 'ZQZQZQZQ' subtitles2016-sample.en" Benchmark 1: rg -c 'ZQZQZQZQ' subtitles2016-sample.en Time (mean ± σ): 80.7 ms ± 1.6 ms [User: 57.7 ms, System: 22.7 ms] Range (min … max): 75.3 ms … 83.3 ms 35 runs
Read the rest
Warning: Ignoring non-zero exit code. Benchmark 2: krep -c 'ZQZQZQZQ' subtitles2016-sample.en Time (mean ± σ): 122.8 ms ± 1.4 ms [User: 372.6 ms, System: 24.4 ms] Range (min … max): 120.2 ms … 125.5 ms 24 runs Summary rg -c 'ZQZQZQZQ' subtitles2016-sample.en ran 1.52 ± 0.03 times faster than krep -c 'ZQZQZQZQ' subtitles2016-sample.en That's a benchmark with no matches, which is the best case essentially for throughput. Now I want to try a benchmark with a high match frequency: $ hyperfine "rg -c 'the' subtitles2016-sample.en" "krep -c 'the' subtitles2016-sample.en" Benchmark 1: rg -c 'the' subtitles2016-sample.en Time (mean ± σ): 411.8 ms ± 3.6 ms [User: 389.7 ms, System: 21.1 ms] Range (min … max): 404.8 ms … 415.7 ms 10 runs Benchmark 2: krep -c 'the' subtitles2016-sample.en Time (mean ± σ): 121.2 ms ± 1.9 ms [User: 364.6 ms, System: 24.9 ms] Range (min … max): 113.2 ms … 123.0 ms 24 runs Summary krep -c 'the' subtitles2016-sample.en ran 3.40 ± 0.06 times faster than rg -c 'the' subtitles2016-sample.en Which is very nice. So I decided to poke at it: $ krep -c the subtitles2016-sample.en Found 29794426 matches $ rg -c the subtitles2016-sample.en 6123710 $ grep -c the subtitles2016-sample.en 6123710 The counts are way off here. At first I thought maybe it was counting every occurrence of `the` instead of every matching line, but when I ask ripgrep to do that, it gets a different answer: $ rg -oc the subtitles2016-sample.en 7739791 $ rg -o the subtitles2016-sample.en | wc -l 7739791 $ grep -o the subtitles2016-sample.en | wc -l 7739791 So not sure what's going on here, but it looks like `krep` might not be giving accurate results. Pushing it a bit more, it seems like it just kind of falls over? $ time rg -c 'You read Sherlock Holmes to deduce that\?' subtitles2016-sample.en 10 real 0.076 user 0.049 sys 0.026 maxmem 923 MB faults 0 $ time krep -c 'You read Sherlock Holmes to deduce that?' subtitles2016-sample.en Found 0 matches real 0.935 user 3.597 sys 0.029 maxmem 918 MB faults 0 I ran the above benchmarks in `/dev/shm` on Linux with an i9-12900K. In terms of the approach here, ripgrep is already using a pretty sophisticated substring search algorithm: https://github.com/BurntSushi/memchr?tab=readme-ov-file#algo... And it uses memory maps (sometimes, when it thinks it will be fast, but it will do so in the single file case on Linux). ripgrep also uses parallelism, but at inter-file level. It sounds like `krep` also uses parallelism, but will use multiple threads when searching a single file. I've considered doing the same in ripgrep, but haven't done enough experiments (or seen enough from someone else) to be convinced that it's the right way to go in general. It might edge out single threaded search in some cases for sure though. EDIT: Looking at the timings in https://dev.to/daviducolo/introducing-krep-building-a-high-p..., I see, for example, ripgrep taking >40 seconds to search for the literal pattern `error` in a 5GB file. Even if you're reading from disk (which the OP is using an SSD), that does not seem right at all. Even for an exceptionally common word like `the` in this haystack, ripgrep can chew through a 13GB file in 5 seconds on my machine: $ time rg -c the full.txt 83499915 real 5.404 user 5.092 sys 0.302 maxmem 12511 MB faults 0 Even if I force reading from disk, we get nowhere near 40 seconds: $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time rg -c the full.txt 83499915 real 10.577 user 5.191 sys 2.105 maxmem 12511 MB faults 42 I'm not saying the benchmark results are definitely wrong, but something looks off here that I can't easily explain. OP, can you please share a way to fully reproduce your benchmark? (Like I did above for `subtitles2016-sample.en`.)
In Show HN: Krep a High-Performance String Search Utility Written in C

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.