encoding_rs

Author of ripgrep here. Maybe, umm, don't depend on auto-detection tools as the ultimate source of whether something is appropriately licensed or not? encoding_rs is clearly licensed, otherwise I wouldn't have used it in ripgrep: https://github.com/hsivonen/encoding_rs Take it up with the auto-detection tools. Or at least do your due diligence before claiming ripgrep is using "unlicensed" code. The Debian copyright notice is correct for ripgrep.

RE: the N/A in ripgrep dependencies specifically, it's a little clearer on my machine (newer version of cargo-license maybe?). Slightly complicated but still above board https://github.com/hsivonen/encoding_rs#licensing > Custom License File (1): encoding_rs

That's not the only use of SIMD in the crate (e.g. see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...), but I haven't looked into exactly where/how it's used further.

As far as I can see it does not use SIMD for UTF8 validation, only likely/unlikely instrinsics, see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...

I'd be curious how this compares with the encoding_rs crate (https://github.com/hsivonen/encoding_rs), which also offers UTF-8 validation (among many other things).

> and supports a wider range of character encodings Since this is an uncommon feature, I'd like to emphasize this. :-) In particular, ripgrep will automatically search UTF-16 encoded files via BOM sniffing. That means you can run ripgrep over a directory on Windows and be confident that it will correctly search both UTF-8 and UTF-16 encoded files automatically without having to think about it. More generally, it supports all encodings found in the Encoding Standard[1]. However, only UTF-16 is automatically detected (where UTF-8 is the presumed default), so you'll need to explicitly specify `-E sjis` (for example) if you want to search Shift_JIS encoded files. Also, I didn't really need to do much of anything to get this working. This is all thanks to @hsivonen's encoding_rs[2] crate, which is now (I think) in Firefox. [1] - https://encodi