hyperscan

Look at a fast regex library [1] and take a subset of those tests or benchmarks and apply to match. [1] https://github.com/intel/hyperscan/tree/master That could be a fun comparison. As someone mentioned, regular expressions are extremely fast, and have been benchmarked forever.

But that's slower for small query sizes. This describes a few algorithms: http://0x80.pl/notesen/2018-10-18-simd-byte-lookup.html Both the alternative version by Geoff Langdale, and the special case for small sets, are substantially similar to the algorithms used in Hyperscan (truffle and shufti). https://github.com/intel/hyperscan Having something hard coded for spaces can be much faster, especially since 5 of the 6 characters are a range: a wrap-around subtraction and an unsigned less-than does the first 5; an equality compare does the other.

Maybe even better examples would be <https://github.com/intel/hyperscan>, <https://github.com/simdutf/simdutf>, and my own <https://github.com/ashvardanian/StringZilla> :)

You can beat CTRE throughput with a non-compile-time solution. My first recommendation will be to look at HyperScan. It has remarkable throughput: https://github.com/intel/hyperscan I’ve only avoided it in the tutorial, as I want to keep the build system lean. I wouldn’t be surprised if it’s 10x faster than Boost in the average case.

Maybe some other PCRE-compatible implementation offers streaming. For instance, https://www.intel.com/content/www/us/en/developer/articles/t... says it has this feature, but of course given who it's from it may be tied to a single brand of CPU. github seems to be https://github.com/intel/hyperscan

This might be relevant here: there is a Rust implementation of this algorithm by BurntSushi [1], and discussion [2] about the performances and Hyperscan [3] in the case of Suricata [4]. HyperScan being regular expression matching library using AMD64 SIMD instructions by Intel. HyperScan is in low maintenance mode, but there is a maintained fork which is compatible with more architectures: Vectorscan [5] [1] https://github.com/BurntSushi/aho-corasick/ [2] https://github.com/BurntSushi/aho-corasick/discussions/136 [3] https://github.com/intel/hyperscan [4] https://github.com/OISF/suricata [5] https://github.com/VectorCamp/vectorscan

I think of WAFs as an extra safety net. Defense in depth. The author complained about the performance cost of WAFs in general, but not all WAFs have be structured like ModSecurity. They could for example be based on something like https://github.com/intel/hyperscan and perf is at a very different level.

Another example of old timey Unix code just breaking things in minor point releases. See https://abi-laboratory.pro/index.php?view=changelog&l=glibc&... and https://github.com/intel/hyperscan/issues/359.

It’s a bespoke scanning setup designed to deal with GitHub’s scale, minimise false positives, and scan fast enough to be in the `git push` request/response cycle. Under the hood it’s using Intel’s hyperscan as the regex engine. https://github.com/intel/hyperscan

The main difference is that binwalk goes through a file linearly, searching for patterns like magic bytes, and tries to extract everything it finds. The problems with this: - very noisy, finding a lot of false positives (license code, format inside another format, etc). - very slow, trying to extract irrelevant things - imprecise, because it finds patterns in the middle of a file, where it's actually not relevant on the first level of extraction unblob solves these problems by being smarter about the file formats, recognizing them by their specification, for example unpacks format header structs and carves out files based the information in the header (size, offset). See a simple example for NTFS [1]. We also went to great lengths preventing unnecessary work by skipping formats inside another [2]. We are using hyperscan [3] instead of gr

The official repo[1] is looking pretty dead. Edit: This is a surprisingly heavy build. Over 15 minutes on my somewhat older i5. I suggest using make -j instead of cmake -build to speed it up. Decent number of compiler warnings too. [1] https://github.com/intel/hyperscan

Intel has an implementation of this technique here as well: https://github.com/intel/hyperscan

It was an Intel project. https://github.com/intel/hyperscan/

I built (or at least "designed most of, built parts of") Hyperscan: https://github.com/intel/hyperscan - a fast multiple regular expression matcher. It was not originally open sourced; it was a commercial product for some time. I wasn't satisfied with what already existed at the time (and neither were our customers) because most regular expression libraries are not focused on performance (using backtracking), streaming (again, backtracking) across multiple inputs - e.g. packets - without holding on to old data - and handling lots of regular expressions at scale. re2 came along while we were still doing a commercial project but had zero impact on our sales, as it didn't handle streaming (weirdly, imo, as it could have) but it also didn't scale very well to large numbers of patterns. I also designed simdjson (https://github.com/simdjson/sim

An interesting project would be to have a go at trying to automatically generate that malicious input. If I remember correctly, re2 has heuristics to see whether construction is "still worth it"; a particularly nasty input would stay just on the verge of "worth doing construction" but cost as much as possible (if you introduce too many novel states re2 will stop doing the construction at all). DFA based engines are often relatively simple under the hood, and backtrackers (like modern JIT'd libpcre) may outrace them due to having optimizations. Hyperscan (https://github.com/intel/hyperscan) a project I worked on for many years avoids the potential DoS attacks on backtrackers or the ones RE2 has, but surely has its own weak spots (notably, it assumes literal factors of the regular expression are rare in the input). We did build some mitigat