tensor2tensor

Discussion about DeepL: https://news.ycombinator.com/item?id=15122764 Attention is not new. Everyone uses it (for translation and many related tasks). It's very much the standard right now. Avoiding recurrent connections inside the encoder or decoder is also not completely new. That came up when people tried to only use convolutions. Googles Transformer was made public in June 2017, in the paper Attention is all you need, https://arxiv.org/abs/1706.03762, including TensorFlow code, https://github.com/tensorflow/tensor2tensor . Note that the new thing here is that they neither use recurrence nor convolution but rely entirely on self-attention instead, with simple fully-connected layers, in both the encoder and the decoder. DeepL directly compares their model to Transformer, in terms of performance (BLEU score), here: https://www.deepl.com/

Not that I know of, but the source code is available to train a Transformer model in a single day. https://github.com/tensorflow/tensor2tensor#walkthrough

> we need better ways of composing this `deep` abstractions Tensor2Tensor[0], from the Google Brain team, has some strong recent results in that direction. Related paper: "One Model To Learn Them All"[1]. [0]: https://github.com/tensorflow/tensor2tensor [1]: https://arxiv.org/pdf/1706.05137.pdf

One Model To Learn Them All Deep learning yields great results across many fields, from speech recognition, image classification, to translation. But for each problem, getting a deep model to work well involves research into the architecture and a long period of tuning. We present a single model that yields good results on a number of problems spanning multiple domains. In particular, this single model is trained concurrently on ImageNet, multiple translation tasks, image captioning (COCO dataset), a speech recognition corpus, and an English parsing task. Our model architecture incorporates building blocks from multiple domains. It contains convolutional layers, an attention mechanism, and sparsely-gated layers. Each of these computational blocks is crucial for a subset of the tasks we train on. Interestingly, even if a block is not cruci