gpt-3

OpenAI publishes a huge amount of research and makes huge amounts of data available for research. They do collaborate with other institutions and researchers. eg: GPT-2: paper https://d4mucfpksywv.cloudfront.net/better-language-models/l... with code: https://github.com/openai/gpt-2, output dataset https://github.com/openai/gpt-2-output-dataset and a bunch of blog posts with all manner of details GPT-3: paper https://arxiv.org/abs/2005.14165 dataset https://github.com/openai/gpt-3 etc etc.

I have doubts it was extensively trained on German data. Who knows about GPT4, but GPT3 is ~92% of English and ~1.5% of German, which means it saw more "die, motherfucker, die" than "die Mutter". (https://github.com/openai/gpt-3/blob/master/dataset_statisti...)

GPT-3 is opensource. GPT-4 isn't. https://github.com/openai/gpt-3

>Certainly it is pleasing that they are not just doing Hindi, but some of these languages must be represented online by a very small corpus of text indeed. I wonder how effectively an LLM can be trained on such a small training set for any given language? as long as it's not the main language it doesn't really matter. Besides English(92.6%), the biggest language by representation (word count) is taken up by french at 1.8%. Most of the languages GPT-3 knows are sitting at <0.2% representation. https://github.com/openai/gpt-3/blob/master/dataset_statisti... Competence in the main language will bleed into the rest.

Doesn't really matter. There's lots of positive transfer in individual language learning. Competence in one language bleeds into competence in others. https://arxiv.org/abs/2108.13349 GPT-3 is fluent in many languages despite English taking up 93% of the corpus by word count. French is next with 1.8% https://github.com/openai/gpt-3/blob/master/dataset_statisti... Dunno the statistics of language presence with GPT-4 but it takes it up another level in terms of its multilingual capabilities.

1.7k forks… https://github.com/openai/gpt-3 apart from that, several pre trained corpuses had been around for a while