KGkgeist·Sep 2, 2024
Calling it a "transformer" is misleading when discussing language modelling because it now means a very specific ML architecture while your project seems to be about Markov chains + hardcoded rules using regexps https://github.com/bennyschmidt/llimo/blob/master/models/Cha...
The idea of tokenizing words and producing completions is not unique to the original transformers, it's a basic idea from NLP. So I'm not sure why you think it should be called a transformer just because it uses tokenized inputs and produces completions as well. It's like saying your new programming language has a "Java-based architecture" simply because they both have classes (and nothing else in common otherwise).
>I didn't want to call it a "computer" or a "completer". Have a better word?
I've seen projects which also use Markov chains + additional rules ontop, for
Read the rest
example there's quite a few projects called "Markov chains with POS tagging":
https://github.com/26medias/context-aware-markov-chains
>not from lookups or assembling based on rules.
Not quite sure about "it's not based on rules" when your code has things like:
const MATCH_FIRST_MODAL = new RegExp(/IS|AM|ARE|WAS|HAS|HAVE|HAD|MUST|MAY|MIGHT|WERE|WILL|SHALL|CAN|COULD|WOULD|SHOULD|OUGHT|DOES|DID/);
or
const properNoun = `${part.value} `;
if (isPrevNNP) {
result += prependArticle(query, properNoun);
}
Pretty sure your examples in the video are also cherry-picked. The very first example is you asking "where is Paris?" What really happens is, one of the hardcoded regexps transforms it to "Paris is" and then the bigram model repeats the second sentence in the Paris dataset verbatim.