icu

> You can't even clearly define what an "atomic sequence of glyphs" is. Kinda. Grapheme cluster breaks are defined in Unicode, but they have all the baggage and edge-cases you'd expect from human languages evolving over time, so they can be encoded in as a few as a thousand rules : https://github.com/unicode-org/icu/tree/main/icu4c/source/da...

ICU: International Components for Unicode: https://en.wikipedia.org/wiki/International_Components_for_U... unicode-org/icu: https://github.com/unicode-org/icu Microsoft/ICU: https://github.com/microsoft/icu IDN: Internationalized domain name: https://en.wikipedia.org/wiki/Internationalized_domain_name Punycode: https://en.wikipedia.org/wiki/Punycode IDN homograph attack: https://en.wikipedia.org/wiki/IDN_homograph_attack CWE-1007: Insufficient Visual Distinction of Homoglyphs Presented to User: https://cwe.mitre.org/data/definitions/1007.html GNU libidn/libidn2: https://gitlab.com/libidn/libidn2 Comparison of regular expression engines > Language features > Part 2; Unicode support: https://en.wikipedia.org/wiki/Comparison_of_regular_expressi...

Apart from the learning experience, there is still value in a small-sized, fast TUI text editor for Linux. However, my two suggestions would be: - add Unicode support (it's the 21st century, so ∑, ®, etc. should finally work); this is not easy, but the earlier you do it the less dramatic the changes will be (check out https://github.com/unicode-org/icu/tree/main/icu4c); - don't allocate the lines individually (as you do now with malloc, having looked at the code); at least use an arena allocator, i.e. allocate larger chunks of memory and the provide your own alloc_line function that uses the larger chunks (called arenas) piecemeal. This will speed things up and reduce fragmentation. A more advanced approach would be not to use per-line buffers but to switch to rope data structures e.g. https://github.com/josephg/librope).

It is surprising how popular is using O(N^2) algorithms that are simple in implementation even in extremely popular libraries. E. g. ICU search method: https://github.com/unicode-org/icu/blob/a7a2fdbcf257fa22c7a2... ICU is used by Python, Windows or OpenOffice.

To all programmers here, the TZ=:<zonefile> syntax is currently unsupported in the icu library (International Components for Unicode): https://unicode-org.atlassian.net/browse/ICU-13694 https://github.com/unicode-org/icu/pull/2213 This affects all packages that have icu as a dependency, one of them being Node.js. https://github.com/nodejs/node/issues/37271 I discovered this the hard way when some code malfunctioned shortly after daylight savings time kicked in.

No, the ICU dictionaries are seen at: https://github.com/unicode-org/icu/tree/main/icu4c/source/da... I assume at compile time it's converted to a more efficient query format

If you're interested in this you should take a look at Chrome's break engine source code for japanese/chinese: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... https://chromium.googlesource.com/v8/v8/+/refs/heads/master/...

Doing a bit more deep diving into the ICU code, it looks like the source code for the Break engine (used by Chinese, Japanese, and Korean) is here: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... and then according to the LICENSE file[1], the dictionary : # The word list in cjdict.txt are generated by combining three word lists # listed below with further processing for compound word breaking. The # frequency is generated with an iterative training against Google web # corpora. # # * Libtabe (Chinese) # - https://sourceforge.net/project/?group_id=1519 # - Its license terms and conditions are shown below. # # * IPADIC (Japanese) # - http://chasen.aist-nara.ac.jp/chasen/distribution.html # - Its license terms and conditions are shown below. # It's interesting to see so

The underlying library is actually using a single dictionary for both Chinese and Japanese https://github.com/unicode-org/icu/tree/7814980f51bca2000a96...