jieba

1. Take a large collection of text, e.g. from https://opus.nlpl.eu/corpora-search/zh-CN&en 2. Split into sentences and tokenize sentences into words, e.g. using https://github.com/fxsjy/jieba 3. Count how often each word appears and sort sentences by descending frequency of the least common word. 4. Use binary search to find a location in the sorted collection of sentences where the difficulty feels about right. Of course this gives you a collection of disjointed sentences, but you can always go to the original file and look at the surrounding context when you find an interesting or confusing one.

Thanks for the kind words! I'm using Jieba[0] because it hits a nice balance of fast and accurate. But I'm initializing it with a custom dictionary (~800k entries), and have added several layers of heuristic post-segmentation. For example, Jieba tends to split up chengyu into two words, but I've decided they should be displayed as a single word, since chengyu are typically a single entry in dictionaries. [0] https://github.com/fxsjy/jieba

The basic strategy is always the same: you use a dictionary to identify possible words, and if there's overlap use a language model to determine the most likely segmentation. The details can be very different, e.g. Fugashi is a wrapper for MeCab https://taku910.github.io/mecab/ which uses conditional random fields, while the most popular Chinese tokenizer is probably Jieba https://github.com/fxsjy/jieba which uses a combination of word frequencies and hidden markov models. For German, the task is much easier, since most words are already separated by spaces, so only compound nouns might be problematic, but it's rare for those to have more than one possible segmentation. (There are of course exceptions, like e.g. Wachstube, which could be either Wach-Stube or Wachs-Tube. EDIT for non-German speakers: cognates are watch-stove = guardroom, w

The segmentation is based on a library called jieba. It seems to work pretty well from my experience. https://github.com/fxsjy/jieba I do notice that some generated words are bogus (compounds of other words or redundant in some way). I have a file that lists redundant words, and when I notice these words I add them to the file so the tool won't generate them again. It also lists the words that they are duplicates of, so that those words can be upranked. This is the file if you're curious: https://github.com/kerrickstaley/Chinese-Vocab-List/blob/mas...