Whisper

> You could say that about some of nvidia's other GPUs too. Technically, yes. Practically, nVidia forbids to deploy their other GPUs in data centers, in the EULA of their drivers: https://www.datacenterdynamics.com/en/news/nvidia-updates-ge... May or may not be enforceable, probably depends on jurisdiction, but I’m pretty sure that EULA gonna scare some people. > People writing pytorch code don't want to think about CUDA vs vulkan at all. Well, there’re also other people. OpenAI’s CEO doesn’t necessarily write pytorch code, but he described the compute costs of ChatGPT as “eye-watering”. > I sincerely hope that tools like openai's triton make it easier to support a variety of GPUs. I’ll believe it when I see it. Performance portability across GPU architectures ain’t trivial. For instance, in my Whisper implementation, the most expensive c

> a lot of new models require the higher-end 80GB A100 variants For training, yeah. But at least in some cases, inference can be optimized to use less memory. For their large Whisper model, OpenAI says they need 10GB of VRAM: https://github.com/openai/whisper#available-models-and-langu... My DirectCompute port of that uses slightly over 4GB of VRAM, see the last column in that table: https://github.com/Const-me/Whisper/blob/master/SampleClips/... I haven’t actually optimized memory usage I only did the DirectCompute port, the memory savings were achieved by Georgi Gerganov in whisper.cpp project, which I ported to Windows.

And there's a fork of that that uses DirectCompute to run it on GPUs without Cuda on Windows: https://github.com/Const-me/Whisper

You could also port that code from nVidia CUDA somewhere else. If you wanna run on servers, Vulkan Compute will probably do the trick. I have ported ML inference to Windows desktops with DirectCompute https://github.com/Const-me/Whisper These NVIDIA A40 GPUs mentioned on the page you’ve linked cost $4000 (Amazon) to $10000 (Dell), deliver similar performance to Intel A770 which cost $350. That’s 1 order of magnitude difference in cost efficiency.

> All they will ever offer is a paid prompt. They sometimes open their older models: https://github.com/Const-me/Whisper

Tensor cores are simply multiplying matrices. Relatively easy to do with a compute shader. Here's an example of such shader in D3D11: https://github.com/Const-me/Whisper/blob/master/ComputeShade...

I’ve asked ChatGPT about hardware. Here’s the response: > As for your specific computer, with 64 GB of RAM and a high-performance GPU like the GeForce 1080 Ti, it should have sufficient resources to run a language model like me for many common tasks. Based on the models open sourced by OpenAI, they are using PyTorch and CUDA. This means their stack requires nVidia GPUs. I think the main reason for their high costs is a single sentence in the EULA of GeForce drivers: https://www.datacenterdynamics.com/en/news/nvidia-updates-ge... It’s technically possible to port their GPGPU code from CUDA somewhere else. Here’s a vendor-agnostic DirectCompute re-implementation of their Whisper model: https://github.com/Const-me/Whisper On servers, DirectCompute is not great ‘coz Windows server licenses are expensive. Still, I did that port alone, and spen