godot-dodo

If you can find a large body of good, permissively licensed example code, you can finetune an LLM on it! There was a similar attempt for Godot script trained a few months ago, and its reportedly pretty good: https://github.com/minosvasilias/godot-dodo I think more attempts havent been made because base llama is not that great at coding in general, relative to its other strengths, and stuff like Starcoder has flown under the radar.

This a continuation of previous work done in the godot-dodo project (https://github.com/minosvasilias/godot-dodo), which involved finetuning LLaMA models on GitHub-scraped GDScript code. Starcoder performs significantly better than LLaMA using the same dataset, and exceeds evaluation scores of both gpt-4 and gpt-3.5-turbo, showing that single-language finetunes of smaller models may be a competitive option for coding assistants, especially for less commonplace languages such as GDScript. The twitter thread also details some drawbacks of the current approach, namely increasing occurences where the model references out-of-scope objects in its generated code, a problem that worsens as the amount of training epochs increases.

The wrapping prompt is also used during inference. (https://github.com/minosvasilias/godot-dodo/blob/f62b90a4622...) Prompting like this is useful for instruct-finetunes, and similar prompts are used by other projects like stanford-alpaca.

On this page: https://github.com/minosvasilias/godot-dodo/tree/main/models It seems that some of the GPT syntax errors appear to be because the models were trained for Godot 3, but the tests were conducted against Godot 4, hence the error messages like "KinematicBody2D does not exist in 4.x (should be CharacterBody2D)".

Thanks for sharing. Why is the training dataset that contains instructions and output wrapped by another enclosing prompt (https://github.com/minosvasilias/godot-dodo/blob/f62b90a4622...) Why does this even work when during inference this wrapping prompt is absent? Wouldnt the model then work best against a inference prompt that follows the wrapping prompt structure, however the desired outcome is to have a model that just works without the wrapping prompt? Edit: see reply from OP, the wrapping prompt is used for inference as well, so misunderstanding on my part

This repository presents finetuned LLaMA models that try to address the limited ability of existing language models when it comes to generating code for less popular programming languages. gpt-3.5-turbo and gpt-4 have proven to be excellent coders, but fall off sharply when asked to generate code for languages other than Python/Javascript etc. The godot-dodo approach to address this: Finetune smaller models on a single one of these languages, using human-created code scraped from MIT-licensed GitHub repositories, with existing GPT models generating instructions for each code snippet. This differs from the dataset generation approach used by projects such as stanford-alpaca or gpt4all, in that the output values of the training set remain high quality, human data, while following the same instruction-following behavior. This will likely pro