mesh-transformer-jax

> With no real knowledge of LLM and only recently started to understand what LLM terms mean, such as 'model, inference, LLM model, intruction set, fine tuning' whatelse do you think is required to make a took like yours? This was mee a few weeks ago. I got interested in all this when FlexGen (https://github.com/FMInference/FlexGen) was announced, which allowed to run inference using OPT model on consumer hardware. I'm an avid user of Stable Diffusion, and I wanted to see if I can have an SD equivalent of ChatGPT. Not understanding the details of hyperparameters or terminology, I basically asked ChatGPT to explain to me what these things are: Explain to someone who is a software engineer with limited knowledge of ML terms or linear algebra, what is "feed forward" and "self-attention" in the context of ML and large language models. Provi

I'm one of the Ray developers, thanks for the shoutout :) If you're curious about how Ray is used for LLMs, here are some interesting examples of LLM projects using Ray! - Alpa does training and serving with 175B parameter models https://github.com/alpa-projects/alpa - GPT-J https://github.com/kingoflolz/mesh-transformer-jax - Another HN thread on training LLMs with Ray (on TPUs in this case) https://news.ycombinator.com/item?id=27731168 - OpenAI fireside chat on the evolution of their infrastructure and usage of Ray for training https://www.youtube.com/watch?v=CqiL5QQnN64 - Cohere on their architecture for training LLMs https://www.youtube.com/watch?v=For8yLkZP5w&t=3s Some other thoughts 1. There is a lot more we want to do to make Ray better for working with large language models and for making training, serving, and batch inference wor

You can skip to step 4 using something like GPT-J as far as I understand: https://github.com/kingoflolz/mesh-transformer-jax#links The pretrained model is already available.

GPT-J specifically was written using Google Cloud's TPU stuff. https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b They even have a google colab notebook thingie. Not much setup needed, just an account. https://colab.research.google.com/github/kingoflolz/mesh-tra...

The most promising are probably one of the smaller versions of bloom https://huggingface.co/docs/transformers/model_doc/bloom or gpt-j https://github.com/kingoflolz/mesh-transformer-jax

They hinted at it in the screenshot, but the goods are linked from the https://6b.eleuther.ai page: https://github.com/kingoflolz/mesh-transformer-jax#gpt-j-6b (Apache 2)

I know nothing, but have heard Hugging Face is in that direction. https://github.com/huggingface/transformers >Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. > These models can be applied on: > - Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages. > - Images, for tasks like image classification, object detection, and segmentation. > - Audio, for tasks like speech recognition and audio classification. --- Also I read about GPT-J, whose capability is comparable with GTP-3. https://github.com/kingoflolz/mesh-transformer-jax But I believe it requires buying or renting GPUs.

doesnt seem the code is there - pretrained models are there. https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b https://huggingface.co/EleutherAI/gpt-j-6B isnt that so ?

175B * 16 bits = 350GB, but it does compress a bit. GPT-J-6B, which you can download at https://github.com/kingoflolz/mesh-transformer-jax, is 6B parameters but weighs 9GB. It does decompress to 12GB as expected. Assuming the same compression ratio, download size would be 263GB, not 350GB.

I’m surprised that no one has answered for three hours! The answer is at https://github.com/kingoflolz/mesh-transformer-jax It has detailed comparisons and a full breakdown of the performance, courtesy of Eleuther.

Hello all, We've spent quite some time benchmarking the best fine-tuning techniques for GPT-J at NLP Cloud (https://nlpcloud.io). Finding the best solution was not straightforward and we had to look at things like speed, server costs, ease of development, accuracy of the fine-tuned model... It took time but we ended up with a nice setup (and we are now officially proposing GPT-J fine-tuning + automatic deployment on our platform). Here are our key takeaways: - The best methodology seems to be the one from the Mesh Transformer Jax team: https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md - Fine-tuning on GPU is not ideal. Even several GPUs used in parallel with Deepspeed can be very slow. We used 4 GPUs Tesla T4 in parallel, and it took 1h30 to only compute our first checkpoint (+ 80GB of RAM used...), for a tr

GPT-J is trained on TPU VM generously offered by Google as part of their TPU research cloud. It’s a small version true but the results are almost on par with reported GPT-3 Source - https://github.com/kingoflolz/mesh-transformer-jax

I hadn't noticed this format before, but wouldn't one want to include a sha or git tag in such a citation? howpublished = {\url{https://github.com/kingoflolz/mesh-transformer-jax}}, I did see the month and year keywords, which I guess would narrow down the number of commits it could have possibly been, but nowhere near the reproducibility of citing the software used

If you’re also curious how it does vs GPT3 here’s the Zero-shot performance evaluations table: https://github.com/kingoflolz/mesh-transformer-jax#zero-shot...