transformers

https://github.com/huggingface/transformers (was) based on PyTorch, so they originally had lot of the models like gpt2-small. Because of that influence, PyTorch was probably going to win there. The one con about TF Serve (TFX) is packaging the entire model into the container (so that ends up being 3gb?+). This was a couple of months ago, so I might be wrong by now ... It was an area I wasn't very confident I could do, TF Serve is really new, so there aren't many guides (and many were already out of date).

That probably wouldn't work here. That s3 bucket is hosting models downloaded at runtime/startup [1] and looks like under normal runs it would be cached. If this is being used in Apple's CI pipeline though the whole thing is being torn down between builds so every build and test has to fetch it again. [1] https://github.com/huggingface/pytorch-transformers/search?q...

"Almost everyone" working on NLP uses one of hugginface's pretrained models at one point or another, sooner or later: https://github.com/huggingface/pytorch-transformers It's so damn convenient, and so nicely done. And they keep doing neat things like this one: https://github.com/huggingface/swift-coreml-transformers Kudos to Julien Chaumond et al for their work!

huggingface offers a whole family of pretrained transformers. https://github.com/huggingface/pytorch-transformers

For deep learning / neural networks, I think these are the best balance of theory and practice (not to mention great visualizations): http://cs231n.github.io More on the intuition/theory side for NNs, but Michael Nielsen’s book is highly recommended: http://neuralnetworksanddeeplearning.com For NLP, you can try this extremely slick interactive course on spaCy (highly recommended Swiss Army knife library for NLP) from package author Ines Montani: https://course.spacy.io The Keras tutorials are solid, here’s one on seq2seq models: https://blog.keras.io/a-ten-minute-introduction-to-sequence-... For more cutting edge NLP, it looks like the fast.ai course covers the Transformer model (the basis of BERT, GPT-2, et al. This is a great overview of that architecture: http://nlp.seas.harvard.edu/2018/04/03/attention.html And PyTorch implementation

"Microsoft will release the code and pre-trained models.", though there is no pointer to where the release will happen. Training gargantuan language models is getting quite expensive, so releasing code + pre-trained models is significant. The architecture is a derivation of PyTorch BERT [0], with an MTL loss function on top. [0] https://github.com/huggingface/pytorch-pretrained-BERT