AMD-SHARK-Studio

I wasn't talking about Llama specifically. I was thinking of the SHARK Stable Diffusion port (which uses MLIR/IREE), as it considerably outpaced the ONNX runtime. But apparently the performance of llama on torch-mlir is progressing: https://github.com/nod-ai/SHARK/issues/1707

We have it running as part of SHARK (which is built on IREE). https://github.com/nod-ai/SHARK/tree/main/shark/examples/sha...

That's a very good question and definitely something of interest. Note, that the compiler is only part of this story (as Mika also mentioned here). With OpenXLA we want to be able to take advantage of the best of what each platform can provide and opsets like StableHLO are meant to provide a portability layer while being expressive enough that targeting specialized hardware efficiently is possible. If you look inside the openxla/iree repo (as well as iree/iree-samples and iree/iree-jax repos, paper Scott cited or sers of IREE like Shark (https://github.com/nod-ai/SHARK#quick-start-for-shark-stable...)) you'll see some example.

Arch is packaging ROCm natively in testing, and PyTorch has official-ish builds of it now (and maybe Arch will build it too). But you should also check out MLIR which, for instance, is used very sucessfully here and is benching way faster than rocm: https://github.com/nod-ai/SHARK/

That speed is kinda eyebrow raising, as the CUDA version can do 1080p in seconds. If you are the dev, have you considered SHARK? Vulkan performance seems to be excellent on AMD cards: https://github.com/nod-ai/SHARK/tree/2f36de319a52479039c7ea0...

Since you mention stable diffusion, there's this: https://github.com/nod-ai/SHARK/blob/main/shark/examples/sha... Looks like it's still early days, but clearly it has recently been made to work and there's an effort going on to make it better.

Try SHARK on your AMD GPUs for SD. Follow the setup here: https://github.com/nod-ai/SHARK/tree/main/shark/examples/sha.... It works with Pytorch -> torch-mlir -> MLIR / IREE -> vulkan. Works on both Windows and Linux. And has a simple gradio web UI https://github.com/nod-ai/SHARK/tree/main/web but we plan to enable better UI integrations very soon. Join us on discord https://discord.gg/RUqY2h2s9u if you have any trouble. Appreciate any / all feedback.