WizardLM

I've been following the state of things, but I'm not sure which ones are the best. There's Meta's CodeLlama[1], Mistral's Codestral[2], DeepSeek AI's DeepSeek-Coder-V2-Instruct[3], CodeGemma[4], Alibaba's CodeQwen[5], and Microsoft's WizardCoder[6]. I'm pretty sure CodeLlama is out of date now. I've heard DeepSeek LLMs are good and DeepSeek-Coder-V2-Instruct was released recently. With the good reputation and its massive size (236b) I'd guess it is the best coding LLM, but if it's not being trained efficiently, maybe Codestral and Codestral Mamba come close. I don't think the best coding LLMs are close to GitHub Copilot but I could be wrong since I'm just relaying information that I've heard secondhand. [1] https://ai.meta.com/blog/code-llama-large-language-model-cod... [2] https://mistral.ai/news/codestral/ [3] https://github.com/deepsee

Deleted. Archive: http://web.archive.org/web/20240415221214/https://wizardlm.g... https://github.com/nlpxucan/WizardLM/commit/4218c0ba64a84cfd...

WizardCoder is probably close to state of the art for open models as of right now. https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder https://huggingface.co/WizardLM/WizardCoder-Python-34B-V1.0 Top of the line consumer machines can run this at a good clip, though most machines will need to use a quantized model (ExLlamaV2 is quite fast). I found a model for that as well, though I haven't used it myself: https://huggingface.co/oobabooga/CodeBooga-34B-v0.1-EXL2-4.2...

This post is misleading, in a way that is hard to do accidentally. - They compare the performance of this model to the worst 7B code llama model. The base code llama 7B python model scores 38.4% on humaneval, versus the non-python model, which only scores 33%. - They compare their instruct tuned model to non-instruct-tuned models. Instruction tuning can add 20% or more to humaneval performance. For example, WizardLM 7B scores 55% on humaneval [1], and I've trained a 7B model that scores 62% [2]. - For another example of instruction tuning, Stablecode instruct tuned benchmarks at 26%, not the 20% they cite for the base model [3] - Starcoder, when prompted properly, scores 40% on humaneval [4] - They do not report their base model performance (as far as I can tell) This is interesting work, and a good contribution, but it's im

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the new project by hugging face. What's are folks' take on this vs Llama 2, which was recently released by Facebook Research? While I haven't tested it extensively, 70B model is supposed to rival Chat GPT 3.5 in most areas, and there are now some new fine-tuned versions that excel at specific tasks like coding (the 'codeup' model) or the new Wizard Math (https://github.com/nlpxucan/WizardLM) which claims to outperform ChatGPT 3.5

WizardCoder-15B (an evol-instruct starcoder fine-tune) is probably the best performing open model atm: https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder

In terms of speed, we're talking about 140t/s for 7B models, and 40t/s for 33B models on a 3090/4090 now.[1] (1 token ~= 0.75 word) It's quite zippy. llama.cpp performs close on Nvidia GPUs now (but they don't have a handy chart) and you can get decent performance on 13B models on M1/M2 Macs. You can take a look at a list of evals here: https://llm-tracker.info/books/evals/page/list-of-evals - for general usage, I think home-rolled evals like llm-jeopardy [2] and local-llm-comparison [3] by hobbyists are more useful than most of the benchmark rankings. That being said, personally I mostly use GPT-4 for code assistance to that's what I'm most interested in, and the latest code assistants are scoring quite well: https://github.com/abacaj/code-eval - a recent replit-3b fine tune the human-eval results for open models (as a point of reference

Your benchmark lacks the current #2 https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder It beats Claude and Bard. You could probably get a 4bit 15B model going in 16GB of RAM and be approaching GPT4 in capability. ...on an old laptop, lol Let's eat OpenAI's lunch! They deserve it for trying to steal this tech by "privatizing" a charity, hiding scientific data that was supposed to be shared with us by said charity whose purpose was to help us all, and dishonestly trying to persuade the government not to let us compete with them.

WizardCoder model Surpasses Claude-Plus (+6.8), Bard (+15.3) and InstructCodeT5+ (+22.3) on HumanEval Benchmarks twitter description: https://twitter.com/WizardLM_AI/status/1669109414559911937 code: https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder

It's not quite so trivial to implement this solution. SL instruction tuning actually needs a lot of examples, and only recently there have been approaches to automate this, like WizardLM: https://github.com/nlpxucan/WizardLM To try my solution, this would have to be adapted to more complex training examples involving quoted text with prompt injection attempts. Similar points holds for RL. I actually think it is much more clean to solve it during instruction tuning, but perhaps we also need some RL. This normally requires training a reward model with large amounts of human feedback. Alternative approaches like Constitutional AI would first have to be adapted to cover quotes with prompt injection attacks. Probably doable, but takes some time and effort, all the while prompt injection doesn't seem to be a big practical issue currently.