fastvalidate-utf-8

op due to validation failures, which don't really matter for benchmarking. Even though the distribution of code points will be roughly half ASCII, for a 32-byte AVX2 vector, the probability of a pure ASCII vector is something less than 2^-32 (since non-ASCII initial bytes translate into more than one byte of output, and the probability of 32 ASCII code points in a row is 2^-32, the probability of 32 bytes in a row is less, by an amount I'd rather not try to calculate). So for this particular benchmark, random UTF-8 should be roughly equivalent to "no ASCII". To verify, I disabled generating ASCII in the linked code, and the numbers came out pretty much identical. If anything, even more ASCII bytes would be a more interesting test, since that would make the quick-ASCII branch less predictable. If you know of any good UTF-8 corpora for common use cases, I'd be happy to benchmark them. Given that different sets of "random UTF-8 bytes" generated with this method will virtually always follow the same path, the exact distribution doesn't really matter when measuring cycles/byte. Input that is purely 2-byte code points will be faster in terms of cycles/code point than purely 4-byte input, but that's mostly a property of UTF-8 being variable length. I doubt there's much speed to be gained by adding any sort of adaptive behavior beyond the ASCII check. Generally these days you want as few branches as you can manage, and this algorithm has only one that matters.