grok-1

Is t5x an encoder/decoder architecture? Some more general options. The Flax ecosystem https://github.com/google/flax?tab=readme-ov-file or dm-haiku https://github.com/google-deepmind/dm-haiku were some of the best developed communities in the Jax AI field Perhaps the “trax” repo? https://github.com/google/trax Some HF examples https://github.com/huggingface/transformers/tree/main/exampl... Sadly it seems much of the work is proprietary these days, but one example could be Grok-1, if you customize the details. https://github.com/xai-org/grok-1/blob/main/run.py

Anyone know the system requirements? Anyone even able to run it? In their last release "grok 1" the issues are full of people who can't even run it: https://github.com/xai-org/grok-1/issues

HN readers might be interested in this URL: https://github.com/xai-org/grok

From the official release page [1]: > We are releasing the base model weights and network architecture of Grok-1, our large language model. Grok-1 is a 314 billion parameter Mixture-of-Experts model trained from scratch by xAI. > This is the raw base model checkpoint from the Grok-1 pre-training phase, which concluded in October 2023. This means that the model is not fine-tuned for any specific application, such as dialogue. > We are releasing the weights and the architecture under the Apache 2.0 license. > To get started with using the model, follow the instructions at github.com/xai-org/grok. A little disappointing they are not releasing the weights for the Grok-1 finetuned model. [1] https://x.ai/blog/grok-os

the submission that doesn't have a paywall: https://news.ycombinator.com/item?id=39737313 or the discussion on the code itself: https://news.ycombinator.com/item?id=39737281 TFA: https://x.ai/blog/grok-os TFC: https://github.com/xai-org/grok-1#readme

This is the weights and the model under Apache 2.0 license. What do you mean by open-source? https://github.com/xai-org/grok/blob/main/model.py https://github.com/xai-org/grok/blob/main/run.py#L25

As promised, weights are available. For those who don't like Twitter / X, Here are the details from the README: - 314B parameter Mixture of Experts model - Base model (not finetuned) - 8 experts (2 active) - 86B active parameters - Apache 2.0 license - Code: https://github.com/xai-org/grok - Weights: https://github.com/xai-org/grok?tab=readme-ov-file#downloadi... Even more open than OpenAI.