trax

Is t5x an encoder/decoder architecture? Some more general options. The Flax ecosystem https://github.com/google/flax?tab=readme-ov-file or dm-haiku https://github.com/google-deepmind/dm-haiku were some of the best developed communities in the Jax AI field Perhaps the “trax” repo? https://github.com/google/trax Some HF examples https://github.com/huggingface/transformers/tree/main/exampl... Sadly it seems much of the work is proprietary these days, but one example could be Grok-1, if you customize the details. https://github.com/xai-org/grok-1/blob/main/run.py

Impressive model, thank you for releasing it under a business-friendly license! Have you considered using Google's sparse "scaling transformer" architecture as the base? Even at 3B scale it can generate 3-4x more tokens per FLOP while being competitive at perplexity with a dense transformer. I think OpenAI uses a variant of it in their ChatGPT-3.5-Turbo product. Here is the paper https://arxiv.org/abs/2111.12763 and the implementation https://github.com/google/trax/blob/master/trax/models/resea... if you are interested. Hope you get to look into this!

Thank you for developing the pipeline and amassing considerable compute for gathering and preprocessing this dataset! I'm not sure if this is the right place to ask about this, but could you consider training an LLM using a more advanced, sparse transformer architecture (specifically, "Terraformer" from this paper https://arxiv.org/abs/2111.12763 and this codebase https://github.com/google/trax/blob/master/trax/models/resea... by Google Brain and OpenAI)? I understand the pressure to focus on training a straightforward LLaMA replication, but of course you see that it's a legacy dense architecture which limits its inference performance. This new architecture is not just an academic curiosity but is already validated at scale by Google, providing 10x+ inference performance boost on the same hardware. Frankly, the community's compute budget

A few more transformer implementations that I’ve found: https://github.com/pbloem/former/blob/master/former/transfor... https://github.com/openai/blocksparse/blob/master/examples/t... https://github.com/google/trax/blob/master/trax/models/trans...

What about Trax (https://github.com/google/trax) and how does it compare with Flax or Haiku? Interesting that googlers who are supposed to use Tensorflow are now actively developing a new autograd engine and at least three new DL frameworks on top of it. What do you think about this segmentation?

That's not a fair comparison because JAX does not really provide high-level libraries (and for ones it does, it introduces many more than those four functions). For example, there must be at least 4 different neural network libraries in JAX. (e.g., https://github.com/google/jax/blob/master/jax/experimental/s..., https://github.com/google/trax).