datasets

I like how this article put it: https://martinalderson.com/posts/huggingface-openai-exploit/ > A final point on this - Hugging Face has an enormous attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams. But yeah, from the way HF described it a Python pickle hole looks possible. Their datasets library uses the pandas.read_pickle() method here: https://github.com/huggingface/datasets/blob/d21c5816d5d1961...

This is my (current) favorite list comprehension: https://github.com/huggingface/datasets/blob/871eabc7b23c27d... Someone was feeling awfully clever that day. (Not that I'm not occasionally guilty myself.)

https://github.com/huggingface/datasets https://github.com/huggingface/transformers

"HuggingFace datasets" is an open source Python package: https://github.com/huggingface/datasets/ And they also have ready-to-use scripts for A LOT of the usual datasets: https://huggingface.co/datasets including LAION 400M and LAION 2B: https://huggingface.co/datasets/laion/laion2B-en

Have a look at the datasets library [1], but as a shortcut, you can just create a file named "my_code.json" in jsonlines format with one line per source file that looks like: {"text": "contents_of_source_file_1"} {"text": "contents_of_source_file_2"} ... And then pass that my_code.json as the dataset name. [1] https://github.com/huggingface/datasets