aitextgen

To train small gpt-like models, there's also aitextgen: https://github.com/minimaxir/aitextgen

> I mean this as person that is actually studying this technology I literally publish open-source packages on how to use this technology. https://github.com/minimaxir/gpt-2-simple https://github.com/minimaxir/aitextgen

Hey HN! I've been lurking for a while now and I've finally created something that I feel is worth sharing. I've called this project "Tensorpedia." At its core, Tensorpedia takes in a title and utilizes it as a prompt for GPT-2 to synthesize the introductory part of a Wikipedia article. The machine learning stuff is written using a wonderful library called aitextgen [0], using Wikipedia's "Vital Articles" as a data set [1]. The server is written in Node, and it uses Redis as an article cache. If you want to read my article about it (for some reason), you can check it out here [2]. I created this project to get more experience with server technologies. While I wouldn't say it's a complicated application, I learned quite a lot from it. Additionally, as I was inspired by all of those this-x-doesn't-exist projects from a while back, this proje

AI text content generation is indeed a legit industry that's still in its nascent stages. It's why I myself have spent a lot of time working with it, and working on tools for fully custom text generation models (https://github.com/minimaxir/aitextgen). However, there are tradeoffs currently. In the case of GPT-3, it's cost and risk of brushing against the Content Guidelines. There's also the surprisingly underdiscussed risk of copyright of generated content. OpenAI won't enforce their own copyright, but it's possible for GPT-3 to output existing content verbatim which is a massive legal liability. (it's half the reason I'm researching custom models fully trained with copyright-safe content)

As someone who works on a Python library solely devoted to making AI text generation more accessible to the normal person (https://github.com/minimaxir/aitextgen ) I think the headline is misleading. Although the article focuses on the release of GPT-Neo, even GPT-2 released in 2019 was good at generating text, it just spat out a lot of garbage requiring curation, which GPT-3/GPT-Neo still requires albeit with a better signal-to-noise ratio. Most GPT-3 demos on social media are survivorship bias. (in fact OpenAI's rules for the GPT-3 API strongly encourage curating such output) GPT-Neo, meanwhile, is such a big model that it requires a bit of data engineering work to get operating and generating text (see the README: https://github.com/EleutherAI/gpt-neo ), and it's unclear currently if it's as good as GPT-3, even when comparing models ap

I used @minimaxir's aitextgen library to create a Twitter bot powered by GPT-2. This bot takes Joe Biden's tweets as input and tries to respond in the first-person voice of Donald Trump. I didn't have any success with fine-tuning on a dataset of Trump's tweets, so I "included" all the information in the prompt to GPT-2 via [magic sauce](https://github.com/charlesjlee/trump_gpt2_bot/blob/main/twee...). It took a lot of fiddling, but I'm satisfied with the final output. Links https://news.ycombinator.com/user?id=minimaxir https://github.com/minimaxir/aitextgen https://github.com/charlesjlee/trump_gpt2_bot

As someone who maintains a package to both make it easy to fine-tune GPT-2 or create your own from scratch (https://github.com/minimaxir/aitextgen), this submission is a good run-through of the technical considerations toward building a GPT-2 model. It's both substantially easier and faster than it was when OpenAI released their paper in 2019, thanks to both Huggingface Transformers and Tokenizers making the architectures more efficient and other companies streamlining the training process and make it more efficient for all parts in the pipeline. You don't need a TPU cluster to train a working GPT-2 model, although it helps (unfortunately TPU support on PyTorch-based training like aitextgen is more fussy). A free GPU on Colab gets you most of the way, especially since you can get now a T4 or a V100 which lets you use FP16.

The GPT-2 weights were released by OpenAI when GPT-2 was released. (https://github.com/openai/gpt-2) Around that time (since no one else was doing it) I released a wrapper to streamline that code and make it much easier to finetune on your own data. (https://github.com/minimaxir/gpt-2-simple) Nowadays, the easiest way to interact with GPT-2 is to use the transformers library (https://github.com/huggingface/transformers), of which I've created a much better library for GPT-2 that leverages it. (https://github.com/minimaxir/aitextgen)

A neat trick I found while working with GPT-2 is that byte-pair encoding is, in itself a compression method. With Huggingface Transformers, encoding/decoding this way is very fast. I've implemented this approach in my aitextgen package (https://github.com/minimaxir/aitextgen/blob/master/aitextgen...) to encode massive input datasets as a uint16 Numpy array; when gzipped on disk, it's about 1/10th of the original data set size. However, the technique in this submission gets about compression to 1/10 w/o the gzipping. Hmm.

Fire plays nice with package entry_points as well. Here's a very minimal CLI for one of my PyPI packages: https://github.com/minimaxir/aitextgen/blob/master/aitextgen...

Likely same issue as here: https://github.com/minimaxir/aitextgen/issues/8 Will take a look.

I talk about deviations from previous approaches in the DESIGN doc (https://github.com/minimaxir/aitextgen/blob/master/DESIGN.md), but to answer the difference between aitextgen and Huggingface Transformers: Model I/O: aitextgen abstracts some of the boilerplate and supports custom GPT-2 models and importing the old TensorFlow models better. Training: Completely different from Transformers. Different file processing and encoding, training loop leverages pytorch-lightning. Generation: Abstracts boilerplate, allowing addition of more utility functions (e.g bolding when printing to console, allow printing bulk text to file). Generation is admittingly not that much different than Transformers, but future iterations will increasingly diverge.