transformers

you can fine-tune in 8bits weights for use in collab or similarly more restrained environments https://github.com/huggingface/transformers/issues/14839 https://huggingface.co/hivemind/gpt-j-6B-8bit by fine-tuning you should be able to attain niche-specific better results than default chatGPT theoretically, if you're lucky and thorough I guess

I know nothing, but have heard Hugging Face is in that direction. https://github.com/huggingface/transformers >Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. > These models can be applied on: > - Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages. > - Images, for tasks like image classification, object detection, and segmentation. > - Audio, for tasks like speech recognition and audio classification. --- Also I read about GPT-J, whose capability is comparable with GTP-3. https://github.com/kingoflolz/mesh-transformer-jax But I believe it requires buying or renting GPUs.

caveat: the server only has a single user. Performance wise, pushing and pulling work as fast as any other git server (Though I don't have any hooks). The UI is snappy, though it can be sluggish when initially importing large projects. Out of the 2-dozen projects its handling, the only issue I encountered was the Pi choking while migrating a large project (github.com/huggingface/transformers): I suspect it was hitting an internal timeout when importing large LFS files (ML models). Fortunately, since it uses bare repos and a simple db schema, I performed the git clone operation manually and updated the SQLite db to enable the "mirror" flag & it's been happily syncing since.

Basically stuff a 32 bit value into an 8 bit value (and lose precision). Apparently it doesn't affect the results significantly. More info: https://github.com/huggingface/transformers/pull/17901

Thank you for sharing the command for finetuning! Is it possible to share your ds_config.json? I tried to finetune the 2B model on A100 (40GB) using your command, but got a CUDA out of memory error. The ds_config I used was the one from huggingface (https://github.com/huggingface/transformers/blob/main/tests/...).

It is possible to fine-tune CodeGen using Huggingface Transformers! Then you'd be able to fine-tune it on your own code and use the resulting model. However, training is more expensive -- you'd need an A6000 or better to train the 6B model. Something like the following should work: deepspeed --num_gpus 1 --num_nodes 1 run_clm.py --model_name_or_path=Salesforce/codegen-6B-multi --per_device_train_batch_size=1 --learning_rate 2e-5 --num_train_epochs 1 --output_dir=./codegen-6B-finetuned --dataset_name your_dataset --tokenizer_name Salesforce/codegen-6B-multi --block_size 2048 --gradient_accumulation_steps 32 --do_train --fp16 --overwrite_output_dir --deepspeed ds_config.json Where run_clm.py is this script: https://github.com/huggingface/transformers/blob/main/exampl... It might be doable to set this up on an AWS machine with a beefy G

A lot of SOTA models seem to be gravitating towards transformer based models. Obviously, I can't speak for the entire field, but you can just go take a look at the most popular HuggingFace repos and see what I mean. They started out focused on language, but because transformers have become so popular, they're expanding into the audio and vision domains quickly. Their library called 'transformers' is, outside of research, most peoples go to high level framework as it largely abstracts away a lot of the boilerplate that writing in pure TF, PyTorch, Jax requires. See: https://huggingface.co/spaces https://github.com/huggingface/transformers