True-Story-of-Pangu

Zhipu were training on GPUs and had already figured out the right method. The gap between Pangu and its competitors was getting bigger and bigger" 3. "In this situation, Wang Yunhe ('the current director of Noah', editor's note) and his small model laboratory took action. They claimed that they inherited and transformed from the old 135B parameters, and through training a short few hundred B of data, the average improvement of various indicators was about ten points. In fact, this was their first masterpiece of applying the shell to the large model. Huawei's laymen led the experts, which made the leaders completely unaware of this nonsense. They only thought that there must be some algorithm innovation. After internal analysis, they actually used Qwen (which is published by Alibaba, editor's note) 1.5 110B for continued training.", "By adding layers, expanding the ffn dimension, and adding some mechanisms from the Pangu pi paper, they gathered about 135B parameters. In fact, the old 135B has 107 layers, while this model has only 82 layers, and the various configurations are also different. After training, the distribution of many parameters of the new 135B of unknown origin is almost exactly the same as that of Qwen 110B. Even the class name of the model code was Qwen at the time, and they were too lazy to even change the name. The subsequent model is the so-called 135B V2. This model was also provided to many downstreams at the time, even including external customers." And that's about it. Also, yeah, the article was indeed a translation from Chinese. The [original post] was written in Chinese, and then got translated it to English by github.com/moonlightelite. That's why it felt odd to read. [original post]: https://web.archive.org/web/20250706034203/https://github.co... After reading the article, I feel this is less of a whistle blowing, more of an attack against Wang Yunhe. That's why there's so much emotional expressions, to (maybe) appeal to Huawei and/or the future employer of this individual. But that's just my personal feelings/hint.