mistral.rs

Hey all! I'm the maintainer of mistral.rs. I just landed support for OpenAI-compatible Agent Skills via a /v1/skills endpoint, and it works with local open models. Until now Skills have basically been locked to closed models, and with the ability to have private, local intelligence becoming increasingly important, but this feature allows you to do XYZ with local models. It's fully compatible with OpenAI's /v1/skills API, so you can drop mistral.rs into your existing code with minimal difficulty. We support the accompanying tools too: /v1/files or input_file for attaching files to your prompts, and mistral.rs also allows models to send generated files back using the OpenAI-compatible method. It's also easier than ever to try mistral.rs: we are including prebuilt binaries for NVIDIA CUDA, Apple Silicon, and CPU! # Linux/Mac > curl --proto

There is also another Mistral implementation: https://github.com/EricLBuehler/mistral.rs Not sure what the difference is, but it seems to be just be overall better received.

how does this compare to https://github.com/EricLBuehler/mistral.rs ?

or with open source tools like llama.cpp - https://github.com/ggerganov/llama.cpp/blob/master/examples/... or mistral.rs - https://github.com/EricLBuehler/mistral.rs/blob/master/docs/... lmstudio and ollama use llama.cpp underneath. cut the middle man

I was asking the same question, turns out mistral.rs [0] has pretty good abstractions in order to not depend and package llama.cpp for every platform. [0] https://github.com/EricLBuehler/mistral.rs

Just trying to stay focused on launching first (https://docs.mixlayer.com) and keeping early customers happy, but would love to open source some of this work. It'd probably be a separate crate from candle. If you haven't checked it out yet, mistral.rs implements some of these things (https://github.com/EricLBuehler/mistral.rs). Eric hasn't done multi-GPU inference yet, but I know it's on his roadmap. Not sure if it helped, but I shared an early version of my llama 3.1 implementation with him.