gpt-2-simple

In 2020, a friend and I did this with our mutual WhatsApp chat history. Except instead we fine-tuned GPT-2 instead. (As was the fashion at the time!) We used this one, I think https://github.com/minimaxir/gpt-2-simple I think it took 2-3 hours on my friend's Nvidia something. The result was absolutely hilarious. It was halfway between a markov chain and what you'd expect from a very small LLM these days. Completely absurd nonsense, yet eerily coherent. Also, it picked up enough of our personality and speech patterns to shine a very low resolution mirror on our souls... ### Andy: So here's how you get a girlfriend: 1. Start making silly faces 2. Hold out your hand for guys to swipe 3. Walk past them 4. Ask them if they can take their shirt off 5. Get them to take their shirt off 6. Keep walking until they drop their shirt Andy: Can I state

Tap the contact's name in WhatsApp (I think it only works on a phone) and at the bottom of that screen there's Export Chat. There was some post-processing needed to get it in this format: Alice: blah John: blah blah Edit: OP's GitHub README has instructions for exporting & preprocessing. --- For finetuning GPT-2 I think we used this thing. I ran it on Google Colab. My friend ran it on his GPU, it should be doable on most modern-ish GPUs. https://github.com/minimaxir/gpt-2-simple I tried doing something with this a few months ago though and it was a bit of a hassle to get running (needed to use a specific python version for some dependencies...), I forget the details sorry!

> I mean this as person that is actually studying this technology I literally publish open-source packages on how to use this technology. https://github.com/minimaxir/gpt-2-simple https://github.com/minimaxir/aitextgen

The GPT-2 weights were released by OpenAI when GPT-2 was released. (https://github.com/openai/gpt-2) Around that time (since no one else was doing it) I released a wrapper to streamline that code and make it much easier to finetune on your own data. (https://github.com/minimaxir/gpt-2-simple) Nowadays, the easiest way to interact with GPT-2 is to use the transformers library (https://github.com/huggingface/transformers), of which I've created a much better library for GPT-2 that leverages it. (https://github.com/minimaxir/aitextgen)

If you want to play with your own GPT-2 API, I put this together and shared it on HN yesterday: https://github.com/cortexlabs/cortex/tree/master/examples/py... You can run that and deploy your own GPT-2 API locally, or in the cloud. I'd also recommend checking out Max Woolf's gpt-2-simple, which is a library that makes it really easy to fine tune GPT-2 with your own text (if you've played the free version of AI Dungeon, this is how they trained their model): https://github.com/minimaxir/gpt-2-simple

With GPT-3 getting so many people interested in NLP, and with OpenAI's recently announced pricing plan putting it out of many people's reach, I thought it might be useful for some to see how easy it is to deploy your own GPT-2 API. This project uses a couple tools: - Cortex: An open source model serving platform I help maintain. https://github.com/cortexlabs/cortex - Hugging Face's Transformers: An open source library for using popular language models, like GPT-2. https://github.com/huggingface/transformers This project uses a vanilla pre-trained GPT-2 and PyTorch. If you want to use TensorFlow/ONNX, that's supported as well ( https://github.com/cortexlabs/cortex/tree/master/examples/te... ). If you want to finetune GPT-2 on your own text (a la AI Dungeon), I'd suggest using gpt-2-simple and deploying with Cortex: https://github.com/minim

I've been following minimaxir's work with GPT-2 for a while - I've tried building things on https://github.com/minimaxir/gpt-2-simple for example - and this looks like a HUGE leap forward in terms of developer usability. The old stuff was pretty good on that front, but this looks absolutely amazing. Really exciting project.

See the notes and Colab notebook in the source project repo: https://github.com/minimaxir/gpt-2-simple I am working on a new text generation package which should be even more simple to use.

She said it was amazingly beautiful and would have been really moved by it had it not been written by an AI. I counter-argued that human curation of AI generated text was a perfectly valid expression of love. I lost :-|. The model generates results that are formatted like love letters with reasonably structured phrases. Verbatim reproduction of training content seems very low. But the sentences are often nonsensical and there's not much context preservation apart from some repetition. However it does generate some quite interesting original phrases: " Mon âme a moins de conversations que le désir " " C'est une joie bien trop violente pour moi " " vous [êtes] comme l'eau d'un pur amour " These phrases can be cherry-picked based on the feeling you're trying to convey and used to compose unique letters influenced by the best french writers i

Yeah, I forked them over onto my github profile. The poetry model is available on Gwern's website. https://www.gwern.net/GPT-2 https://github.com/odomojuli/slack-gpt2-poetry You can train your own GPT-2 bot using minimaxir's GPT2-simple https://github.com/minimaxir/gpt-2-simple

I have actually done what they have done in the paper. The primary difficulty that I had is that when attempting to fool people they would basically believe nearly anything that the system would say. Its relatively easy to reproduce see http://trumptwitterarchive.com/archive and then you just follow the directions on https://github.com/minimaxir/gpt-2-simple

About GPT-2 finetuning: I have a Python package which makes the process straightforward (https://github.com/minimaxir/gpt-2-simple), along with a Jupyter notebook to streamline it further (https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRN...).

For finetuning GPT-2 on custom text, my gpt-2-simple package (https://github.com/minimaxir/gpt-2-simple) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text compared to the 345M which is slow even on a GPU. How exactly the large GPT-2 models are deployed is a mystery I really wish was open-sourced more.

Hey HN! As a side project, I used the GPT-2 OpenAI NLG model and trained it on Donald Trump speeches. You can type a sentence and it will generate a few paragraphs that sound like Trump completing the thought. I used https://github.com/minimaxir/gpt-2-simple and some other stuff to re-train, with Starlette being the python web server. I'll write up a blog post later on what I learned and issues I faced. The text generation is pretty slow right now (~20 secs), but I didn't optimize anything since this was just a fun side project. Would love thoughts/feedbacks/funny texts :-)

> Do you think there's any other architecture that would've had better results -while still not taking too long to train? You could easily finetune a pretrained GPT2 model on your lovecraft dataset : https://github.com/minimaxir/gpt-2-simple