DALLE2-pytorch

Thanks to the amazing @lucidrains there's already an open-source implementation of DALL-E 2: https://github.com/lucidrains/DALLE2-pytorch and a pretrained model for it should be released within this year. The same person is also at work on an open-source implementation of Google's Imagen which should be even better (and faster) than DALLE-2: https://github.com/lucidrains/imagen-pytorch. This is possible because the original research papers behind DALLE-2 and Imagen were both publicly released.

In the unCLIP/DALL-E 2 paper[0], they train the encoder/decoder with 650M/250M images respectively. The decoder alone has 3.5B parameters, and the combined priors with the encoder/decoder are the in the neighborhood of ~6B parameters. This is large, but small compared to the name-brand "large language models" (GPT3 et. al.) This means the parameters of the trained model fit in something like 7GB (decoder only, half-precision floats) to 24GB (full model, full-precision). To actually run the model, you will need to store those parameters, as well as the activations for each parameter on each image you are running, in (video) memory. To run the full model on device at inference time (rather than r/w to host between each stage of the model) you would probably want an enterprise cloud/data-center GPU like an NVIDIA A100, especially if runnin

LAION is working on open source alternatives. There's a lot of activity in their discord and they have amassed the necessary training data but I am uncertain as to whether they have obtained the funding needed to deliver fully trained models. Phil Wang created initial implementations of several papers including imagen and parti in his GitHub account. EG: https://github.com/lucidrains/DALLE2-pytorch

As far as I know, the actual trained system is proprietary, and you can only use it by requesting access to their online system for generating imagery: https://openai.com/dall-e-2/ There are open-source efforts to implement it and make trained models available, but I don't imagine they are yet at the same scale of ingested data / model size as OpenAI's system: https://github.com/lucidrains/DALLE2-pytorch

Is it non-reproducible though? Google did not release their model but they did publish their method, which they didn't have to do. People like lucidreams are already at work replicating them so everyone can benefit: https://github.com/lucidrains/DALLE2-pytorch

Check https://github.com/multimodalart/majesty-diffusion or https://github.com/lucidrains/DALLE2-pytorch There is a Google Colab workbook that you can try and run for free :) This is the image-text pairs behind: https://laion.ai/laion-400-open-dataset/

That only lasts until the community copies the paper and catches up. For example the open source DALLE-2 implementation is coming along great: https://github.com/lucidrains/DALLE2-pytorch

There are people working on reproducing the models, see here for Dall-E 2 for example: https://github.com/lucidrains/DALLE2-pytorch It's often not worth it to decentralize the computation of the trained model though but it's not hard to get donated cycles and groups are working on it. Don't fret because Google isn't releasing the API/code. They released the paper and that's all you need.

Allegedly, based on info from a DALL-E 2 paper, it takes about 100k-200k GPU hours to train the model: https://github.com/lucidrains/DALLE2-pytorch/issues/22#issue...

Not a complete answer to your question but you may find this discussion interesting: https://github.com/lucidrains/DALLE2-pytorch/discussions/10 Inference cost and scale seems to be much more favourable than large language models (for now).

They are now starting to train the model. https://wandb.ai/laion/diffusion-prior?workspace=user- https://github.com/lucidrains/DALLE2-pytorch/issues/23

It seems to be empty: https://github.com/lucidrains/DALLE2-pytorch/blob/main/dalle...