dictomaton

Say you want to compute all strings of length 5 that the automaton can generate. Conceptually the nicest way is to create an automaton that matches any five characters and then compute the intersection between that automaton and the regex automaton. Then you can generate all the strings in the intersection automaton. Of course, IRL, you wouldn't actually generate the intersection automaton (you can easily do this on the fly), but you get the idea. Automata are really a lost art in modern natural language processing. We used to do things like store a large vocabulary in an deterministic acyclic minimized automaton (nice and compact, so-called dictionary automaton). And then to find, say all words within Levenshtein distance 2 of hacker, create a Levenshtein automaton for hacker and then compute (on the fly) the intersection between the Lev

Also related: Levenshtein automata -- automata for words that match every word within a given Levenshtein distance. The intersection of a Levenshtein automaton of a word and a DAWG gives you an automaton of all words within the given edit distance. I haven't done any Java in years, but I made a Java package in 2013 that supports: DAWGs, Levenshtein automata and perfect hash automata: https://github.com/danieldk/dictomaton

My Java library, which does some of the things described in the article (DAWGS, perfect hashing, Levenshtein automata): https://github.com/danieldk/dictomaton

This very topic was discussed yesterday. For a given word, you can construct an automaton that matches that word within a certain edit distance in O(n) where n is the length of the word. Since the result is a DFSA, you can also see if a string matches in O(n). (See the work of Schulz & Mihov.) Moreover, rather than going over each word in a dictionary to find out whether that word is within a certain distance, you can compute the intersection language of the Levenshtein automaton and the dictionary automaton (which is generally faster, intersection is O(mn) in the number of states of the two automata). For an implementation of this and more, see: https://github.com/danieldk/dictomaton

> besides finite state dictionaries, ..., String mappings based on perfect hash automata I'm implementing a similar library in Rust. One of the key things I've found missing in Daciuk's great work on the topic is the handling of the registry. In your code[1], it looks like you're using a straight a hash table. I thought you'd like to know that I've had great success sacrificing complete DFA minimization in exchange for a small constant memory overhead by using an LRU cache of only some classes of states. My code still needs a good deal of polish, and I intend to write a blog post, but I bet you can make sense of the general strategy from the code and reuse it.[2] N.B. Lucene's implementation permits sacrificing complete minimization, but they don't use an LRU cache and I think still permit the registry to grow without bound. [1] - https:/

FWIW, my Dictomaton Java library also has support for Levenshtein automata (besides finite state dictionaries, perfect hash automata, and String mappings based on perfect hash automata): https://github.com/danieldk/dictomaton

Semi-offtopic: The are more compact representations, e.g. you can store the dictionary in a deterministic acyclic minimized finite state automaton (which can be stored in a flat array/slice). This gives you O(n) time lookups, where n is the length of the word, and reduces much of the redundancy in a dictionary. Words within an edit distance can be found by computing a Levenshtein automaton for the word (which can be done in linear time) and computing the intersection language of the dictionary automaton and the Levenshtein automaton. This approach is fast and very compact. I have a Java implementation: https://github.com/danieldk/dictomaton

Shameless plug: I have a Java library that does this even more efficiently. It constructs a Levenshtein automaton of a word in linear time and then computes the intersection between a dictionary (stored in a deterministic acyclic minimal finite state automaton) and the Levensthein automaton. It also has other features, such as: - Perfect hash automata (an automaton that gives a unique hash for every word/sequence). - String-keyed maps that use a finite state automaton to store keys. More info: https://github.com/danieldk/dictomaton http://search.maven.org/#artifactdetails%7Ceu.danieldk.dicto...

The article takes a very weird turn: If we need extra information about the words, we can use an additional data structure along with the MA-FSA. We can store it in a hash table. This is not necessary at all, since an acyclic finite state automaton can be used as a perfect hash function. If you sum the right language cardinalities of the transitions followed, you get a unique hash code (1..n, where n is the number of strings in the automaton). For efficiency, you can precompute the cardinalities in the transitions or states. You can then use a normal array for whatever data you want to associate with a word. With respect to parts 1 & 2 of the series, one can construct a Levenshtein automaton for a given word in linear time[1]. Such an automaton matches any string within the given edit distance of the word. You can then compute the interse

Or store the lexicon in a determinisitic acyclic finite state automaton. E.g. (shameless plug): https://github.com/danieldk/dictomaton Though, having implemented a language guesser myself, it's only an issue with very short texts (a few words). On longer texts models based on character n-grams achieve very high accuracies.

And my implementation in Java :): https://github.com/danieldk/dictomaton

Especially considering that there are incremental algorithms for constructing the minimized DFA. For instance, I use one such algorithm in my library that (among other things) uses a minimized DFA to store String sets in Java: https://github.com/danieldk/dictomaton (It also supports perfect hashing automata and Levenshtein automata.)

Sure, in Java you don't have to deal with GADTs. But you do have to deal with the fact that people think functions are dark magic and expect you to extend an abstract class instead. The amount of type wizardry that Haskell permits is far larger than Java and C++ teaches us that in practice you have to expect all possible combinations language features that are allowed. A random Java library will have a far more understandable API than a random Haskell package, since there are not so many permutations of language features possible. If everybody stuck to just Haskell 98, things would be far simpler. Besides, have you ever tried to use the equivalent of Parsec or QuickCheck in Java? As a matter of fact, I just committed QuickCheck-like unit tests five minutes ago [1] :): https://github.com/danieldk/dictomaton/commit/7aaa89aae68524... I write