libu8ident

On Unicode TR39 https://unicode.org/reports/tr39/ (secure identifiers). https://github.com/rurban/libu8ident/ Also on perfect hashes: https://github.com/rurban/nbperf, also gperf and cmph improvements.

Note that this table is of extremely bad quality and can NOT be used in production without patches. See https://github.com/rurban/libu8ident/#confusables

That's why unicode published the security guidelines and mechanisms to avoid such attacks. In 2004 already. The problem is that nobody cared. Browsers invented punycode instead of following tr39, email ditto. But ok, at least something. Java did it, cperl did, rust did it. Everybody else is vulnerable. Esp. most other programming languages, filesystems and login systems. https://github.com/rurban/libu8ident/blob/master/doc/c11.md

You don't need a new definition, you just need to follow the official Unicode security guidelines. I recommend the Moderate Restrictive Security profile for identifiers for mixed scripts. TR39. Plus allow Greek with Latin. This way you can identify Cyrillic, Greek, CFK or any recommended script, but are not allowed to mix Cyrillic with Greek. And you still can write math with Greek Symbols. What we don't have are a standard string library to compare or find strings. wcscmp does not do normalization. There is no wcsfc (foldcase) for case insensitivity. There's no wcsnorm or wcsnfc. I'm maintaining such a library. coreutils, diff, grep, patch, sed and friends all cannot find Unicode strings, they have no string support. They can only mimic filesystems, finding binary garbage. Strings are so rthi g different than pure ASCII or BINARY garbag

A filesystem accepting only NFD should be filed as bug. They can normalize it internally to NFD, as Apples previous HFS+ did. But even worse than that is Python's NFKC, which normalizes ℌ to H and so on. The recommended normalizations are NFC for offline normalization (like in compiled languages and databases) and NFD for online, where speed trump's space. unicode.org talking that much about NFKC was a big mistake. NFKC is crazy and doesn't even roundtrip. The whole TR31 XID_Start/Continue sets are mostly because of NFKC issues, not so about stability. But people bought it for its stability argument. I'm just writing a library and linter for such issues: https://github.com/rurban/libu8ident Also note that C++23 will most likely enforce NFC identifiers only. Same problem as with this filesystem. My implementation was to accept all normal.