https://github.com/facebookresearch/metaseq/blob/main/projec...
I really enjoyed reading the log book from the training of OPT-175B at Meta… I guess it’s all classified info but it’d be fun to read a blog post about the crazy day to day issues you run into when doing things at this scale :)
In Beam: Reflection's 501B open-weight modelfacebookresearch
metaseq
Python2022–2026
facebookresearch / metaseq · Archived
metaseq
Repo for external large-scale work
30Distinct people
31Mentions
0Explicit recommenders
6.5KGitHub stars
What to know before exploring
What it is
Repo for external large-scale work
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year31 dated mentions
Observed mentions: 2022-05-03 02:04:25 UTC to 2026-10-05 23:51:10 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 17 | 6 | 11 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 11 | 1 | 10 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 2 | 1 | 1 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 1 | 0 | 1 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
Metaseq
A codebase for working with Open Pre-trained Transformers, originally forked from fairseq.
Community Integrations
Using OPT with 🤗 Transformers
The OPT 125M--66B models are now available in Hugging Face Transformers. You can access them under the facebook organization on the Hugging Face Hub
Using OPT-175B with Alpa
The OPT 125M--175B models are now supported in the Alpa project, which
enables serving OPT-175B with more flexible parallelisms on older generations of GPUs, such as 40GB A100, V100, T4, M60, etc.
Using OPT with Colossal-AI
The OPT models are now supported in the Colossal-AI, which helps users to efficiently and quickly deploy OPT models training and inference, reducing large AI model budgets and scaling down the labor cost of learning and deployment.
Using OPT with CTranslate2
The OPT 125M--66B models can be executed with CTranslate2, which is a fast inference engine for Transformer models. The project integrates the SmoothQuant technique to allow 8-bit quantization of OPT models. See the usage example to get started.
Using OPT with FasterTransformer
The OPT models can be served with FasterTransformer, a highly optimized inference framework written and maintained by NVIDIA. We provide instructions to convert OPT checkpoints into FasterTransformer format and a usage example with some benchmark results.
Using OPT with DeepSpeed
The OPT models can be finetuned using DeepSpeed. See the DeepSpeed-Chat example to get started.
Getting Started in Metaseq
Follow setup instructions here to get started.
Documentation on workflows
Training
API
Background Info
Background & relationship to fairseq
Chronicles of training OPT-175B
Support
If you have any questions, bug reports, or feature requests regarding either the codebase or the models released in the projects section, please don't hesitate to post on our Github Issues page.
Please remember to follow our Code of Conduct.
Contributing
We welcome PRs from the community!
You can find information about contributing to metaseq in our Contributing document.
The Team
Metaseq is currently maintained by the CODEOWNERS: Susan Zhang, Naman Goyal, Punit Singh Koura, Moya Chen, Kurt Shuster, David Esiobu, Igor Molybog, Peter Albert, Andrew Poulton, Nikolay Bashlykov, Binh Tang, Uriel Singer, Yuchen Zhang, Armen Aghajanya, Lili Yu, and Adam Polyak.
License
The majority of metaseq is licensed under the MIT license, however portions of the project are available under separate license terms:
Megatron-LM is licensed under the Megatron-LM license
Read the full README on GitHubWhat Hacker News says
31 sourcesTraining LLM from scratch is a super important issue that affects the pace and breadth of iteration of AI almost as much as the raw hardware improvements do. The blog is fun but somewhat shallow and not technical or very surprising if you’ve worked with clusters of GPUs in any capacity over the years. (I liked the perspective of a former googler, but I’m not sure why past colleagues would recommend Jax over pytorch for LLMs outside of Google.) I hope this newco eventually releases a more technical report about their training adventures, like the PDF file here: https://github.com/facebookresearch/metaseq/tree/main/projec...
In Training LLMs from ground zero as a startupNevermind, I figured it out: https://github.com/facebookresearch/metaseq/blob/main/projec...
In Model training diary/journal for LLMs?Very much agree... I think people underestimate how finicky these models are to train. So much so that one of the big announcement around GPT-4 was the fact that OpenAI found a way to make it smoother and more predictable.
As evidence by this 114 page log of how engineers resolved problems that came up during the training of OPT-175, which lasted for several months. https://github.com/facebookresearch/metaseq/blob/main/projec...
In Consortium launched to build the largest open LLMThere is so much "secret sauce" that we're unconsciously relying upon. I learned a lot of secret sauces by reading through various implementations, "logbooks" [0] by various researchers, and so on. It definitely makes AI/ML feel more like an art than a science, and it makes it very difficult to re-implement in another language (e.g., Julia) or framework.
0: Such as https://github.com/facebookresearch/metaseq/blob/main/projec...
In The open source learning curve for AI researchersThe number of people who can actually pre-train a true LLM is very small.
It remains a major feat with many tweaks and tricks. Case in point: the 114 pages of OPT175B logbook [1]
[1] https://github.com/facebookresearch/metaseq/blob/main/projec...
In See the pitch memo that raised €105M for four-week-old startup Mistral> It's not only free as in beer, but free as in libre
If by it you mean LLaMA 1, then I don't think per the license one can use it for commercial projects. So, it isn't really libre. That said, all indications are that LLaMA 2 would be fully FLOSS.
If you meant OPT, it isn't libre either: https://github.com/facebookresearch/metaseq/blob/main/projec...
In Meta's plan to offer free commercial AI models puts pressure on Google, OpenAIStable Diffusion will run on any decent gaming GPU or a modern MacBook, meanwhile LLMs comparable to GPT-3/ChatGPT have had pretty insane memory requirements - e.g., <https://github.com/facebookresearch/metaseq/issues/146>
In Visual ChatGPTIt seems so, check this log book from Meta: https://github.com/facebookresearch/metaseq/blob/main/projec...
In Ask HN: Is There On-Call in ML?Meta OPT-175B LLM training logbook [pdf]
1 points · Open discussionThanks for laying out the plan. I was trying to understand the cost of each of these steps below and started wondering about the following:
> rough steps:
> 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $.
Pile seemed quite clean and manageable to me (I was able to preprocess it ~8 hours for a simple task on consumer grade hardware). Is Pile clean and rich enough for LLM training too ?
> 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts.
It seems like the standard instructGPT model itself is based on a 1 billion param GPT model. Wouldn't that fit on a 24GB RTX 3090 ? Might take longer, maybe not enough opportunity for hyper-paramet
Read the rest
er search, but still possible right ? Or is hyper-parameter search on a thousand machines in parallel the real magic sauce here ?
> 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec...
Sounds like a good opportunity to learn. No pain, no gain :-)
> 4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see OPT-IML https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 to get a sense of the work involved here.
Maybe somebody would open source the equivalent datasets for this soon ? Otherwise the data collection seems prohibitively expensive for somebody trying to do this for fun: contract expert annotators, train them, annotate/reannotate for months ?
rough steps:
1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $.
2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts.
3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec...
4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see
Read the rest
OPT-IML https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 to get a sense of the work involved here.
This is my impression as well. Here is an example of the engineering capabilities required to train the modern models:
https://github.com/facebookresearch/metaseq/blob/main/projec... [PDF]
It's a 114 page document detailing the months of full-time work for multiple engineers that went into training a 175B language model at meta AI.
In Ask HN: Can you crowdfund the compute for GPT?I work in this field (PhD candidate), and what you say is true for smaller models, but not GPT-3 scale models. Training large scale models involved a lot more, as the OP said. It's not just learning rate schedulers, it's a whole bunch of stuff.
See this logbook from training the GPT-3 sized OPT model - https://github.com/facebookresearch/metaseq/blob/main/projec...
In The GPT Architecture, on a NapkinShowing the 15 most recent of 31 sources. Enable JavaScript to search and browse every indexed mention.