smollm

Maybe I can do one better... For the 'create_chat_completion' on this line: https://github.com/simonw/llm-smollm2/blob/afed0963b3a8bcca1... I changed it to: completion = model.create_chat_completion( messages=messages, temperature=0.2, top_p=0.9, presence_penalty=0.06, frequency_penalty=0.5, repeat_penalty=1.0, ) I took as many parameters as I could from their documentation: https://github.com/huggingface/smollm/tree/main/text#transfo... And then added some to reduce repeating. I then used the prompt "Write a 10 line poem on computers." (note the fullstop), to get: In the realm of computation, where magic and science intertwine, The art of programming and the art of computing. Computers are machines that can think, they are computers. They can do what humans cannot – to create.

Might be worth updating the title to "SmolLM: state-of-the-art small language model trained on open datasets" (See the first table of https://huggingface.co/blog/smollm for benchmarks) It was fascinating digging into this to find their dataset weights defined in a declarative YAML file [2]. 70% is from FineWeb/Commoncrawl but filtered using a classifier trained on Llama-70b's rating from 0-5 of the educational content of the text [3]. This is something we know small models like Phi-3 have been doing for a while, but it's great to see a fully open reproduction of it that beats their benchmarks. Definitely supports the idea you can get even better reasoning at smaller model sizes by carefully filtering and curating your training data (and generating good synthetic data from/distilling bigger models). You can see the 450k Llama educational v