a-pretrainers-guide

ce.co/datasets/conceptofmind/cot_submix_ori... NIv2 submix: https://huggingface.co/datasets/conceptofmind/niv2_submix_or... Dialog submix: https://huggingface.co/datasets/conceptofmind/dialog_submix_... You can find the original FLAN repository and all of Shayne Longpre's incredible work here: https://github.com/google-research/FLAN/tree/main/flan/v2#do... Be sure to also read through Shayne's paper on the FLAN collection to get better insight into how the data was created: https://arxiv.org/abs/2301.13688 We are going to soon be releasing a massive causal language modeling dataset containing hundreds of GBs of high-quality instruction data. Look out for that release in the near future. Our work on an open reproduction of FLAN V2 and related projects is all thanks to the generous sponsorship by CarperAI and StabilityAI. You can find out more about CarperAI here: https://carper.ai/ And StabilityAI here: https://stability.ai/ A big thank you to Jason Phang and Fabrizio Milo for helping build the dataset as well. You can find Jason Phang's twitter here: https://twitter.com/zhansheng And Fabrizio Milo's here: https://twitter.com/fabmilo You can check out Shayne's new paper on building pre-training datasets here: https://github.com/shayne-longpre/a-pretrainers-guide/blob/m... This is not an official Google or StabilityAI product. If you have any questions about the data be sure to reach out and ask! I will try to respond promptly: https://twitter.com/EnricoShippole