intel

hyperscan

C++2016–2026

intel / hyperscan

hyperscan

High-performance regular expression matching library

Libraries & SDKs

Explore on GitHub Read Hacker News discussions

18Distinct people

49Mentions

0Explicit recommenders

5.5KGitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

High-performance regular expression matching library

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

rust-lang/regex · linked in 2 of the same discussions
simdjson/simdjson · linked in 2 of the same discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 10:01:16 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year49 dated mentions

Observed mentions: 2016-06-11 17:00:05 UTC to 2026-02-26 18:15:49 UTC.

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
2016	2016-01-01 00:00:00 UTC	2017-01-01 00:00:00 UTC	8	0	8
2017	2017-01-01 00:00:00 UTC	2018-01-01 00:00:00 UTC	7	0	7
2018	2018-01-01 00:00:00 UTC	2019-01-01 00:00:00 UTC	4	1	3
2019	2019-01-01 00:00:00 UTC	2020-01-01 00:00:00 UTC	5	0	5
2020	2020-01-01 00:00:00 UTC	2021-01-01 00:00:00 UTC	9	0	9
2021	2021-01-01 00:00:00 UTC	2022-01-01 00:00:00 UTC	4	0	4
2022	2022-01-01 00:00:00 UTC	2023-01-01 00:00:00 UTC	2	0	2
2023	2023-01-01 00:00:00 UTC	2024-01-01 00:00:00 UTC	4	0	4
2024	2024-01-01 00:00:00 UTC	2025-01-01 00:00:00 UTC	2	0	2
2025	2025-01-01 00:00:00 UTC	2026-01-01 00:00:00 UTC	3	0	3
2026 (partial)	2026-01-01 00:00:00 UTC	2027-01-01 00:00:00 UTC	1	0	1

What Hacker News says

9 sources

glangdale·Nov 24, 2020

This is a cool paper. I'm building a SMT-based superoptimizer for SIMD sequences that is still pretty stupid (it can only handle "lanewise" SIMD i.e. some op on each lane, it can only do AVX2, it can only handle arbitrary graphs up to 4 instructions before it gets too slow). And yet... it concocted a good chunk of a PSHUFB based pattern matcher (effectively "shufti" from https://github.com/intel/hyperscan) that I invented in 2009. If I could push further on this and scale from 4 instructions handled in about 45 minutes (single core, though it scales well) up to 5 instructions (currently takes a month of single-core search time) to 6-7 instructions (I don't even know!), this system would be able to come up with the full version of shufti, a "quite commercially relevant" SIMD sequence that I invented in 2009. Not saying that sequence was th

Read the rest

e cleverest thing ever, but it seemed novel enough, and to have a machine just spew it out when given the black box description of what it's meant to achieve... it would be pretty cool. All I need is a few orders of magnitude worth of search optimizations. That's OK - when I started, my superoptimizer took a couple days to handle 4 instructions, so it might still happen.

In Superoptimizer – A Look at the Smallest Program [pdf] (1987)

glangdale·Aug 6, 2020

Francois is a smart guy, but there's a range of opinions on the matters he raises. We argue a good deal on Twitter ("constructive confrontation" between former Intel Principal Engineers :-) ). He put together the video pretty quickly so don't dismiss it just for lack of polish. Personally, I argue with him a lot on AVX-512. I think AVX-512 is a Good Thing (or will be shortly - the first instantiation in Skylake Server - SKX - isn't great). The biggest meta-problem with AVX-512 is that due to process issues, the pause button got hit just as it appeared in a server-only, 'first cut' form. AVX2 had downclocking issues when it first appeared too - but these were rapidly mitigated and forgotten. I personally feel that SIMD is underutilized in general purpose workloads (partly due to Intel's fecklessness in promoting SIMD - including gratuitous

Read the rest

fragmentation - e.g. a new Comet Lake machine with Pentium branding doesn't even suport AVX2 and has it fused off). Daniel Lemire and I built simdjson (https://github.com/lemire/simdjson) to show the power of SIMD in a non-traditional domain, and a lot of the benefits of Hyperscan (https://github.com/intel/hyperscan) were also due to SIMD. Notably, both use SIMD in a range of ways that aren't necessarily all that amenable to the whole idea of "just do it all on an accelerator/GPU" - choppy mixes of SIMD and GPR-based computing. Not everything looks like matrix multplication, or AI (but I repeat myself :-) ). AVX-512 is a combination of fascinating and frustrating. I enthuse about recent developments on my blog (https://branchfree.org/2019/05/29/why-ice-lake-is-important-...) so I won't repeat why it's so interesting, but the learning curve is steep and there's plenty of frustrations and hidden traps. Intel collectively tends to be more interested in Magic Fixes (autovec) and Special Libraries for Important Benchmarks rather than building first-rate material to improve the general level of expertise with SIMD programming (one of my perpetual complaints). You more or less have to teach yourself and feel your way through the process of getting good at SIMD. That being said, the benefits are often huge - not only because SIMD is fast, but because you can do fundamentally different things that you can't do quickly in GPR ("General Purpose Register") land (look, for example, at everyone's favorite instruction PSHUFB - you can't do the equivalent on a GPR without going to memory).

In What's wrong with Intel, and how to fix it: Former principal engineer unloads

glangdale·Jul 6, 2020

Given how things turned out, this might be a good lesson as to whether "I measured my own library informally on some benchmarks that I couldn't be bothered to describe to you and it WINS!" is trustworthy in future. When we introduced simdjson, we went out of our way to explain (a) why we thought it might be faster than other libraries and (b) how people could replicate our measurements or find fault in our methodology. There were zero performance claims, public or private about the performance of simdjson before this material was available. In my long experience, undocumented and non-reproducible performance claims suffer from the fact that people generally stop benchmarking when they get the result that they wanted to hear, even if the absolute numbers involved are unreasonable. It's very easy to make mistakes in performance measurement

Read the rest

(compiling simdjson with anything short of -O2 or -O3, doing repeated runs of a tiny input and thus getting perfect branch prediction behavior), but people generally fix these mistakes only on the side that makes their stuff look good. Back when I worked on the Hyperscan project ( https://github.com/intel/hyperscan ), we had an API call that allocated scratch memory (necessary due to Hyperscan's rather constrained approach to memory usage). I lost count of the number of times we had users run this allocation (which only ever needed to be done once per thread, effectively) inside their measurement loop. And whenever we were competing against an incumbent system and the people doing the measurement were the authors of that incumbent system, it was amazing how quickly they'd accept the "truth" of their benchmark run.

In A Journey building a fast JSON parser and full JSONPath, Oj for Go

glangdale·Jun 16, 2020

It's entertaining that there is yet another area where "running large numbers of regular expressions, known in advance and not changing all that often" over lots of input is performance critical. This was a key driver behind writing Hyperscan (https://github.com/intel/hyperscan) which I used to work on when it was primarily used for network security (thousands or tens of thousands of regular expression rules for firewalls), but doing this for ad blocking seems pretty sensible too.

In How the most popular Chrome extensions affect browser performance

glangdale·May 10, 2020

Mishit is censored too... I like the number of HN commenters who tried out their favorite Scunthorpe variations. As somehow who designed an implementation of \b for an NFA-based regex matcher (https://github.com/intel/hyperscan) I always have an appreciation for the Scunthorpe problem; we did see quite a number of scam/spam/bad-word detection regex patterns that tried to avoid it with word boundary stuff.

In Scunthorpe Sans – a self-censoring font

glangdale·Mar 29, 2020

This wasn't really on DRAM in the same way as the paper is, interesting though that project was. Sadly, it's more or less defunct. The academic research at UVa has largely come to an end and Micron spun out the project to a startup that seems to be moribund (http://www.naturalsemi.com/). I tracked these guys because we were doing automata in s/w (the https://github.com/intel/hyperscan project at Intel). There were some aspects of their methodology I didn't like (notably, they tended to run benchmarks that spewed matches, slowing Hyperscan down, while turning off matches on the h/w card!) but I found them v. interesting and they had some really thought-provoking stuff.

In ComputeDRAM: In-Memory Compute Using Off-the-Shelf DRAMs (2019) [pdf]

glangdale·Mar 1, 2020

Any good literal search algorithm could do a one-off search for a single long literal 'needle' way faster than the roughly 1GB/s that the author attained with grep. A single string of that length is extremely easy to search for with a range of different algorithms - I would be surprised if a decent approach couldn't keep up with memory bandwidth (assuming your 22GB file is already, somehow, in memory). The mechanics of simply reading such a big file are likely to dominate in practice. We implemented some SIMD approaches in https://github.com/intel/hyperscan that would probably work pretty well (effectively a 2-char search followed by a quick confirm) for this case. Of course, that begs the question - presupposing that any kind of whole-text search is actually the answer to this question. The end result - assuming that you really do have m

Read the rest

ore than a few searches to do - of keeping the results in any kind of prebuilt structure - is way superior to an ad hoc literal search.

In A journey to searching Have I Been Pwned database in 49μs

glangdale·Feb 24, 2020

iirc, Github uses (used?) my old project (https://github.com/intel/hyperscan) at Intel. It's probably faster than the alternatives, although if you want to support all types of regex you'll need to use Hyperscan as a prefilter for a richer regex engine like PCRE. This project looks like it pulls literal factors out of the regex that I type in, maybe to an index a la that Russ Cox blog post a while back about Code Search. It seems to Not Like things that have very open-ended character classes (e.g. \w) unless there is a decent length literal involved somewhere. It seems to have a very rudimentary literal extraction routine, as it decides to give a partial result set when fed an alternation between two literals that it handles pretty well on their own.

In Show HN: Search code in GitHub repos using regular expressions

glangdale·Jan 31, 2020

Oddly, lookbehinds are evil only in a specific backtracking world. We never got around to implementing arbitrary lookarounds in Hyperscan (https://github.com/intel/hyperscan) but if we had done something in the automata world to handle lookaround, lookbehinds are way easier than lookaheads. To handle a lookbehind, you really only need to occasionally 'AND' together some states (not an operation you would normally do in a standard NFA whether Glushkov or Thompson). To handle lookaheads... well, it gets ugly.

In Show HN: Regex Cheatsheet