imagen-pytorch

That's already been done: https://github.com/lucidrains/imagen-pytorch

Depending on how you're measuring it, video generation will come in less than 12 months, or it's already here. - In the Video Diffusion paper[1] they generate 64x64 16 frame videos, model is not released, but an open source implementation using an imagen-like pipeline exists[2], no model is available publicly. - The CogVideo paper[3], which is essentially a huge transformer model, can generate 480x480 videos, the code for the paper and models are open source[4], but be warned that you need a huge GPU like an A100 to run the damned thing. The future of text to video generation will probably be video diffusion, i.e. using 3D UNets, or more likely a much more optimized version of a UNet. Improvements on diffusion models are happening on a daily basis, and we're probably a couple papers away from having a breakthrough in efficiency that we ca

Thanks to the amazing @lucidrains there's already an open-source implementation of DALL-E 2: https://github.com/lucidrains/DALLE2-pytorch and a pretrained model for it should be released within this year. The same person is also at work on an open-source implementation of Google's Imagen which should be even better (and faster) than DALLE-2: https://github.com/lucidrains/imagen-pytorch. This is possible because the original research papers behind DALLE-2 and Imagen were both publicly released.

I'm honestly surprised that they trained a StyleGAN. Recently, the Imagen architecture has been show to be both easier in structure, easier to train, and even faster to produce good results. Combined with the "Elucidating" paper by NVIDIA's Tero Karras you can train a 256px Imagen* to tolerable quality within an hour on a RTX 3090. Here's a PyTorch implementation by the LAION people: https://github.com/lucidrains/imagen-pytorch And here's 2 images I sampled after training it for some hours, like 2 hours base model + 4 hours upscaler: https://imgur.com/a/46EZsJo * = Only the unconditional Imagen variant, meaning what they show off here. The variant with a T5 text embedding takes longer to train.

This implementation popped up on hacker news not too long ago. I got it working on Colab first, and then my own GPU at home. But just barely. Need more memory :) https://github.com/lucidrains/imagen-pytorch

This blog is by the author of Tortoise-TTS (stylised as TorToiSe): https://github.com/neonbjb/tortoise-tts Very cool work being done in ML these days, both inside large corps, and also by independent researchers/"hobbyists" (quotation marks because there is some really expert work being produced by them). And from yesterday, an open-source version of Google's Imagen by lucidrains: https://github.com/lucidrains/imagen-pytorch

It seems like lucidrains is currently working on an implementation [1] of it. I would love it. [1] https://github.com/lucidrains/imagen-pytorch