AITemplate

VoltaML is a relatively vanilla diffusers-based backend, so its not a hairy monster to hack like you may have seen with SAI-based UIs (like Comfy, Fooocus and Automatic) The AITTemplate code is a lightly modified version of Facebook's example dynamic AIT script, to get rid of small issues like VRAM spikes: https://github.com/facebookincubator/AITemplate/tree/main/ex... InvokeAI is also diffusers based, but they seem to mess with the pipeline a bit more. Anyway, all that may be better as a reference for interesting features rather than a backend to try and adopt.

RTX 3080-10GB should work. You could check https://github.com/facebookincubator/AITemplate/tree/main/ex..., and https://www.reddit.com/r/StableDiffusion/comments/xv7m89/met....

AITemplate only supports fp16 data types with fp16 or fp32 accumulation right now. We are working on supporting more data types and quantization. We don't have an official comparison between AITemplate and tvm / onnx for now, but we do have perf numbers like https://github.com/facebookincubator/AITemplate/tree/main/ex..., https://github.com/facebookincubator/AITemplate/tree/main/ex.... Feel free to run these examples on other frameworks and compare perf.

You could check "AITemplate optimizations" section in the blog (https://ai.facebook.com/blog/gpu-inference-engine-nvidia-amd...), and https://github.com/facebookincubator/AITemplate#more-about-a.... The basic idea is to do aggressive kernel fusions.

No, you don't need to train the model differently to use it with AITemplate. Here is an intro example to do inference with AITemplate with a very simple PyTorch model: https://facebookincubator.github.io/AITemplate/tutorial/how_.... For more advanced examples, check out https://github.com/facebookincubator/AITemplate/tree/main/ex...

AITemplate-PyTorch Stable Diffusion is the fastest stable diffusion inference solution by pushing image generation below one second on A100 (batch 1: 0.7s / 25 steps, 1.3s / 50 steps; batch 3: 1.6s / 25 steps, per image 0.55s; batch 16 7.9s / 25 steps, per image 0.49s) for the first time, 2.57X faster than Keras' XLA-based GPU compilation solution. More benchmark numbers and repro at: https://github.com/facebookincubator/AITemplate/tree/main/ex...

tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate. AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion. AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on various ResNet / Bert / VIT models. AITemplate uses sophisticated fusion techniques to optimize perf, including vertical, horizontal, and memory fusions. btw, I'm one of the authors of AITemplate, happy to answer any questions.