mistral-inference

There are many different inference libraries and it's not clear which ones a small company like mistral should back yet IMO. They do release high quality inference code, ie https://github.com/mistralai/mistral-inference

Great work! I love your videos; they've taught me so much. Any plans for a Mixture of Experts (MoE) video? My understanding is that starting from GPT4 most advance models use MoE to some extent. For example, can I take the model from your GPT2 video and just change the feed forward layer to an MoE layer like the one found here (1)? I guess I can just try it myself but I enjoy the expert guidance you provide in your videos. Please don't stop! great content! 1. https://github.com/mistralai/mistral-inference/blob/main/src...

Yes actually, you can use Mistral 7B v0.3 Instruct for this, here’s a guide in the readme, scroll down https://github.com/mistralai/mistral-inference Make sure to put a json schema as well as few shot examples of json values for the same schema in your prompt

> Instead, it turns out a few hundred lines of Python is genuinely enough to train a basic version! actually its not just a basic version. Llama 1/2's model.py is 500 lines: https://github.com/facebookresearch/llama/blob/main/llama/mo... Mistral (is rumored to have) forked llama and is 369 lines: https://github.com/mistralai/mistral-src/blob/main/mistral/m... and both of these are SOTA open source models.

Mistral is not llama architecture. https://github.com/mistralai/mistral-src

I was getting into LLM and I pick up some projects. I tried to dive into the code to see what is secret sauce. But the code is so short to the point there is nothing to really read. https://github.com/facebookresearch/llama I then proceed to check https://github.com/mistralai/mistral-src and suprsingly it's same. What is exactly those codebases? It feels like just download the models.

Well the checkpoint __is__ the computational graph. The graph is also all the code. But if you want it in python... that's here[0]. Please be clear, we keep asking. What are you asking for? Datasets? Training algo? What? Comparing it to software artifacts isn't a good comparison when any program with open source code (visible or free to use) is equivalent to what's being given. You have everything you need to use, edit, and fuck around with. You don't have the exact scheme, but I'll put it this way, if you gave me the hardware I could produce a LLM of high quality from scratch using their architecture. [0] https://github.com/mistralai/mistral-src/blob/main/mistral/m...

The source used to create the model is here: https://github.com/mistralai/mistral-src/blob/main/mistral/m... The source used to train information into the model is elsewhere. You have the engine. You don't have the source of the data.

Hey, how did you try it on M1? I don't see any MPS support. https://github.com/mistralai/mistral-src/issues/2 edit: nevermind https://ollama.ai

It would be very surprising. Mistral AI's github page has more information on their sliding window attention method to achieve this performance: https://github.com/mistralai/mistral-src If Mistral 7b lives up to the claims, I expect these techniques will make their way into llama.cpp. But I would be surprised if the required updates were quick or easy.