bert

Disclaimer, I work on Gemma and open models at Deepmind and the opinions here are my own There were open models from EleutherAI (GPT-Neo), Google Brain (T5X, Bert), and HuggingFace was promoting open models (and others doing open work I haven't listed here) all prior to 2023 and the big Chatgpt moment. https://github.com/EleutherAI/gpt-neo/releases https://github.com/google-research/bert https://github.com/google-research/t5x If you're learning about AI models it's still worthwhile to review these models and codebases because they continue to be the basis of the technology that's being produced today! It'll give you a good perspective of how things have changed, similar to say learning about propeller planes before moving onto modern jet engines.

Apple is behind. People forget that Google was shipping mobile-scale transformer-based LLMs in 2019: https://github.com/google-research/bert By the time Apple has an AI-native product ready, people will already associate it with dehumanization and fascism.

FWIW Google Bert has been doing sentiment analysis with LLMs using text classifiers for more than 7 years at this point: https://github.com/google-research/bert It's definitely a neat solution, but a pretty old one at this point. Those AWS Mechanical Turk employees lost their job a half decade ago, chances are.

OpenAI was most certainly not the first organization to develop (or distribute) an LLM model: https://github.com/google-research/bert https://arxiv.org/abs/1810.04805 Failing that, Seq2seq predates OpenAI's incorporation: https://google.github.io/seq2seq/ https://github.com/google/seq2seq https://arxiv.org/abs/1409.3215

> Has Google/Alphabet publicly released any of their AI models yet? You mean NLP field changing models from Google like BERT [1]? or Transformers paper [2]? or T5 model [3] (used by company doing ChatGPT like search currently on the front page on HN)? 1. https://arxiv.org/abs/1810.04805 code+models: https://github.com/google-research/bert 2. https://arxiv.org/abs/2112.04426 3. https://arxiv.org/abs/1910.10683 code+models: https://github.com/google-research/text-to-text-transfer-tra...

> resulting in large programs with lots of boilerplate That was what I was trying to say when I said "the code required to implement the challenges is large enough that they are considered too inconvenient to use". This makes sense to me. Thank you for this benchmark! I'll probably switch to spyql now from jq. > So, orjson is part of the reason why a python-based tool outperforms tools written in C, Go, etc and deserves credit. Yes, I definitely think this is worth mentioning upfront in the future, since, IIUC, orison's core uses Rust (the serde library, specifically). The initial title gave me the impression that a pure-Python json parsing-and-querying solution was the fastest out there. A parallel I think is helpful to think about is saying something like "the fastest BERT implementation is written Python[0]". While the linked implement

" BERT is a method of pre-training language representations, meaning that we train a general-purpose "language understanding" model on a large text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering). BERT outperforms previous methods because it is the first unsupervised, deeply bidirectional system for pre-training NLP." https://github.com/google-research/bert

This video explains a legendary paper, BERT. It leverages the Transformer encoder and comes up with an innovative way to pre-training language models (masked language modeling). BERT has a significant influence on how people approach NLP problems and inspires a lot of following studies and BERT variants. Code https://github.com/google-research/bert (TensorFlow) https://github.com/huggingface/transformers (PyTorch)

GPT-2 https://openai.com/blog/gpt-2-1-5b-release/ https://talktotransformer.com/ https://github.com/openai/gpt-2 (GPT-2 docker image needs a version tweak to get it working) and BERT: https://github.com/google-research/bert

> BERT returns a tensor of dimension (n_tokens, hidden_size) [1]. Where hidden size has no relationship with the vocabulary "BERT returns" is ambiguous here. During pretraining last layer is loggits for one hot vocab vector, the same as in GPT: https://github.com/google-research/bert/blob/master/run_pret...

Maybe I am not understanding your point. Out of the box, given a sequence of n tokens, BERT returns a tensor of dimension (n_tokens, hidden_size) [1]. Where hidden size has no relationship with the vocabulary. You can then fine-tune a model on this representation to do various tasks, e.g. sentiment classification. Thus BERT is said to be a language representation model. Out of the box, given a sequence, GPT-2 returns a distribution over the vocabulary [2] from which you can draw to find the most likely next word. Thus GPT-2 is said to be a language generation model. You could of course play with the masking token of BERT call it recursively to force BERT to generate something, and you could chop off some layers of GPT-2 to get some representation of your input sequence, but I think that is a little past the original question. [1] https://

The project mentioned (BERT) in the article is really interesting, you should look into it. https://github.com/google-research/bert

I don't think you could get access to the actual models that are being used to run e.g. Google Translate, but if you just want a big pretrained model as a starting point, their research departments release things pretty frequently. For example, https://github.com/google-research/bert (the multilingual model) might be a pretty good starting point for a translator. It will probably still be a lot of work to get it hooked up to a decoder and trained, though. There's probably a better pretrained model out there specifically for translation, but I'm not sure where you'd find it.