transformers

The GPT-2 weights were released by OpenAI when GPT-2 was released. (https://github.com/openai/gpt-2) Around that time (since no one else was doing it) I released a wrapper to streamline that code and make it much easier to finetune on your own data. (https://github.com/minimaxir/gpt-2-simple) Nowadays, the easiest way to interact with GPT-2 is to use the transformers library (https://github.com/huggingface/transformers), of which I've created a much better library for GPT-2 that leverages it. (https://github.com/minimaxir/aitextgen)

Just a random example I encountered the other day, take a look at this: https://github.com/huggingface/transformers/blob/master/note... It’s a tutorial on how to use some NLP ML library. It’s a pretty excellent example of where this format shines, imho, and as you can see it’s absolutely literate. Btw I also identify as primarily a developer, but over the past few years been confronted with these notebooks more and more and I’ve learned to appreciate their use over time. But as with all good things, they’re sometimes overused.

Try spacy alpha 3.0 it integrate the https://github.com/huggingface/transformers library You should almost always use XLnet large in order to achieve the best accuracy

This video explains a legendary paper, BERT. It leverages the Transformer encoder and comes up with an innovative way to pre-training language models (masked language modeling). BERT has a significant influence on how people approach NLP problems and inspires a lot of following studies and BERT variants. Code https://github.com/google-research/bert (TensorFlow) https://github.com/huggingface/transformers (PyTorch)

With GPT-3 getting so many people interested in NLP, and with OpenAI's recently announced pricing plan putting it out of many people's reach, I thought it might be useful for some to see how easy it is to deploy your own GPT-2 API. This project uses a couple tools: - Cortex: An open source model serving platform I help maintain. https://github.com/cortexlabs/cortex - Hugging Face's Transformers: An open source library for using popular language models, like GPT-2. https://github.com/huggingface/transformers This project uses a vanilla pre-trained GPT-2 and PyTorch. If you want to use TensorFlow/ONNX, that's supported as well ( https://github.com/cortexlabs/cortex/tree/master/examples/te... ). If you want to finetune GPT-2 on your own text (a la AI Dungeon), I'd suggest using gpt-2-simple and deploying with Cortex: https://github.com/minim

For those interested in related/alternative approaches, one or more of the following established open-source libraries might appeal to you: - Snorkel (training data curation, weak supervision, heuristic labeling functions, uncertainty sampling, relation extraction): https://github.com/snorkel-team/snorkel - AllenNLP (many pretrained NLP research models for tasks beyond text classification, model training and serving, visualization/interpretability utilities): https://github.com/allenai/allennlp - Spacy (tokenization, NER/POS + tagging visualizer, pretrained word vectors, integration with DL models): https://github.com/allenai/allennlp -huggingface Transformers (latest and greatest pretrained models, e.g. BERT): https://github.com/huggingface/transformers - ...or a barebones “from scratch” solution in less than an hour with a Colab noteboo

As someone who has work with all three NLP toolkit: huggingface, openmt-py and fairseq. I always have trouble juggling through the heavy abstraction of openmt-py. For example in openmt-py you need to write fields, reader and raw datasets before you even load into their complex dataset class. Each item is heavily abstracted through several layers of classes. I understand this improve code reuse, but introduce a huge steep curve for newcomer. Huggingface approach on the other hand is slightly more "messy" [2] but easier to understand and add your own tweak. [1] https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/bin/p... [2] https://github.com/huggingface/transformers/tree/master/src/...

The model is open sourced on GitHub: https://github.com/openai/image-gpt Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch, and it has dependency hell so it's not easy to create a wrapper tool for it. Since it's still the GPT-2 architecture, it might be possible to port the weights to Huggingface Transformers (for the RGB generation), and then write a wrapper to extend it for the image rendering. (filed an issue here: https://github.com/huggingface/transformers/issues/5088 )

Exciting! It's been asked to be integrated into the Hugging Face transformers library already: https://github.com/huggingface/transformers/issues/4658

Graph NNs aren't really used for SOTA NLP tasks. On the other hand, larger and larger transformer networks are constantly improving. Assuming you are in the Northern Hemisphere so "last summer" means Julyish, then Google's T5[1] and Microsoft Turing-NLG[2] come to mind. I find keeping an eye on HuggingFace's models list[3] is useful for this. [1] https://arxiv.org/abs/1910.10683 [2] https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1... [3] https://github.com/huggingface/transformers#model-architectu...

This is cool - might be worth training a simple discriminator model to identify your utterances, and then you can use the plug-and-play language model (PPLM - https://github.com/huggingface/transformers/blob/master/exam...) to generate utterances modeling a specific speaker without special tokens. Could also take less time to fine-tune.

Paper here: https://arxiv.org/abs/1909.11942 Really impressed at the speed with which Hugging Face ported this to their transformers library -- Google released the model and source code Oct 21 [1] and it was available in the library just 8 days later [2]. [1] https://github.com/google-research/google-research/commit/b5... [2] https://github.com/huggingface/transformers/commit/c0c208833...