pythia

You can grep for bad words. What you can't do(unless hoops are jumped through) is to verify that weights came from the same dataset. You can set the same random seed and still get different results. Calculations are not that deterministic. (https://pytorch.org/docs/stable/notes/randomness.html#reprod...). >I am overall skeptical that this is true in the case of LLMs This skepticism seems reasonable. EleutherAI have documentation to reproduce training (https://github.com/EleutherAI/pythia#reproducing-training). So far I haven't seen it leading to anything. Lots of arxiv papers I've seen complain about time and budget constraint even regarding finetunes, forget pretraining.

The ones I saw mentioned so far were Flan, Cerebras, GPT-J, and RWKV. Not yet mentioned: * Pythia https://github.com/EleutherAI/pythia * GLM-130B https://github.com/THUDM/GLM-130B - see also ChatGLM-6B https://github.com/THUDM/ChatGLM-6B * GPT-NeoX-20B https://huggingface.co/EleutherAI/gpt-neox-20b * GeoV-9B https://github.com/geov-ai/geov * BLOOM https://huggingface.co/bigscience/bloom and BLOOMZ https://huggingface.co/bigscience/bloomz

Is this the link? https://engineering.fb.com/2019/05/21/ai-research/pythia/ or maybe this? https://github.com/EleutherAI/pythia I'm confused.