llimo

example there's quite a few projects called "Markov chains with POS tagging": https://github.com/26medias/context-aware-markov-chains >not from lookups or assembling based on rules. Not quite sure about "it's not based on rules" when your code has things like: const MATCH_FIRST_MODAL = new RegExp(/IS|AM|ARE|WAS|HAS|HAVE|HAD|MUST|MAY|MIGHT|WERE|WILL|SHALL|CAN|COULD|WOULD|SHOULD|OUGHT|DOES|DID/); or const properNoun = `${part.value} `; if (isPrevNNP) { result += prependArticle(query, properNoun); } Pretty sure your examples in the video are also cherry-picked. The very first example is you asking "where is Paris?" What really happens is, one of the hardcoded regexps transforms it to "Paris is" and then the bigram model repeats the second sentence in the Paris dataset verbatim.

chatting with the bot about Paris? Is that nothing more than a standard Markov chain? Nothing else going on? Again the answer is an obvious "No" it's a chat bot - what about the NLP work, or the chat interface, etc. makes you ask if it's nothing more than a standard [insert vague philosophical idea]? To me, your question is like when people were asking if jQuery "is just a monad"? I don't understand the significance of the question - jQuery is a library for web development. Maybe there are some similarities to this philosophical concept "monad"? See: https://stackoverflow.com/questions/10496932/is-jquery-a-mon... It's like saying "I looked at your website and have concluded it is nothing more than an Array."

ie), call it a Markov model or "chain", or "just lookups", anything you like - but it's more specifically than all those a language model. Embeddings: Where my embeddings differ is that they're kept in a structure reflective of their source context. That's how I know what the meaning of the word is. OpenAI's embedding is like this: Text: "The dog" Embedding: { "object": "embedding", "index": 0, "embedding": [ -0.006929283495992422, -0.005336422007530928, -4.547132266452536e-05, -0.024047505110502243, ... ], } And those values, particularly those for "dog" will be - like you said - computable with others to find things like similarity, etc. And using that math, they can tell you what the next word will be - and probably a lot of other useful chat-specific things I haven't gotten into. My method is also commonly used in machine learning, and would be TF-IDF vectorization except that I don't fully vectorize the semantics, because it's just based on frequency and token structure: { "the": { "dog": { "was": 834, "is": 721, "will": 36 "has": 442 } } } This system predicts "was" as the next word because it usually is the next word after "dog" (in the source data). This library was built to ultimately provide completions, not have a conversation, so no doubt OpenAI's approach works better for chat. I am however already making a chat model. Here's my approach if anyone cares: The completer already gives great completions and fast, but some of them make no sense to what was asked. The chat model I'm working on here (https://github.com/bennyschmidt/llimo/pull/1) can just get all completions and use parts-of-speech codes to match a completion to the cursor. I don't have this fully implemented yet, but you can get the idea in this PR. This is like an NLP layer specific to chat - has nothing to do with the next-token prediction in general, and there are no NLP libraries in `next-token-prediction` (the npm). The example I've been using to explain this is: User: "Where is Paris?" Convert to cursor: "Paris is" Get completions: "in France", "really nice", "a first name", etc. Analyze input: "Where is" refers to a location Analyze completions: "in France" refers to a location Answer: "Paris is in France" I hope it makes sense! Thanks for your comment. Edit: And yes, again, the library does not claim to "be an LLM" - it's not that at all, it's actually a next-token prediction library that has methods like `getTokenPrediction` etc. Though you are set up to build an LLM with this if you train enough data :) The library is not going to provide a `chat` function like GPT, not going to provide an img2img API like Stable Diffusion, that would be whatever chat-based or image-based LLM is being made with this library.

ow which one is the best suggestion: "Paris is in France." "Paris is nice this time of year." "Paris Hilton liked my post!" "Paris was my favorite city overall." In this case, "is" is still the best because it scores higher than "Hilton" and "was" - because it follows "Paris" more frequently. So in addition to the text file of billions of sentences, I need to make a ranking system to give a point score to every possible word in the file at every possible position it might be in. To make it all faster (because a super massive text file is not feasible), the billions of sentences are not actually represented in a single text file, but as a deeply nested JavaScript object our computers are more optimized to traverse and lookup, along with the point values for each word. At this point, _with enough sentences on file_ with ranked suggestions and fast lookups, you can complete almost anything a user could input. This is what I shared today. > good chat interface To put the sentence-finisher to use as a chat bot, all you need to do is convert the user's question or "prompt" into the beginning of a sentence. For example: "Where is Paris?" -> "Paris is" and let the completer give you "in France". So the answer is: "Paris is in France." Does this make sense?

hmidt/llimo Thanks for the comment and motivation to write about it!