unicode-segmentation

I made a JavaScript library for handling unicode characters, by porting Rust's high-quality library, [unicode-segmentation](https://github.com/unicode-rs/unicode-segmentation). Hopefully will be a good alternative in environments where the standard `Intl.Segmenter` API is not available.

It isn't made clear in the repo but what this is doing [1] is a small amount of code to glue SQLite's tokenizer API to the Rust unicode-segmentation crate [2] which in turn is using the TR-29 algorithm with generated tables [3]. [1] https://github.com/signalapp/Signal-FTS5-Extension/blob/main... [2] https://unicode-rs.github.io/unicode-segmentation/unicode_se... [3] https://github.com/unicode-rs/unicode-segmentation/blob/mast... International Components for Unicode is the reference library for this and other Unicode algorithms. https://en.wikipedia.org/wiki/International_Components_for_U...

In general, unicode requires you think differently about strings depending on context. Here's my rule of thumb. 1. If you are transporting a unicode string, reading/writing over the network or to a file, think in terms of UTF-8 bytes. Do not attempt to splice the string, treat it as an atomic unit. 2. If you are parsing a string, think in terms of code points (runes in Go, chars in Rust). A good example would be the Servo CSS parser. [1] 3. If you're comparing/searching/inspecting/sorting a string in code, segment by grapheme clusters and normalize, then do what you came to do. [2] 4. If you're displaying a string, think in terms of pixels. Do not attempt to limit a string by length in "characters" (nee grapheme clusters in the unicode world) but rather measure by what the renderer does with the string. Each character can be a thoroughly