cog

Founder of Replicate here. We open pull requests on models[0] to get them running on Replicate so people can try out a demo of the model and run them with an API. They're also packaged with Cog[1] so you can run them as a Docker image. Somebody happened to stumble across our fork of the model and submitted it. We didn't submit it nor intend for it to be an ad. I hope the submission gets replaced with the upstream repo so the author gets full credit. :) [0] https://github.com/Vaibhavs10/insanely-fast-whisper/pull/42 [1] https://github.com/replicate/cog

I'm in the same situation. I found this cog project to dockerise ML https://github.com/replicate/cog : you write just one python class and a yaml file, and it takes care of the "CUDA hell" and deps. It even creates a flask app in front of your model. That helps keep your system clean, but someone with big $s please rewrite pytorch to golang or rust or even nodejs / typescript.

the cog template is just starter code to make it super simple to deploy llama-v2 on any infrastructure of your choosing! More about cog https://github.com/replicate/cog Our thinking was just that a bunch of folks will want to fine-tune right away, then deploy the fine-tunes, so trying to make that easy... Or even just deploy the models-as-is on their own infra without dealing with CUDA insanity!