cog

They have a docker image, I would start with that. It's pretty easy to setup a container in the cloud these days. They even have a tool to customize the container: https://github.com/replicate/cog

As mentioned in sibling comments, Torch is indeed the glue in this implementation. Other glues are TVM[0] and ONNX[1] These just cover the neural net though, and there is lots of surrounding code and pre-/post-processing that isn't covered by these systems. For models on Replicate, we use Docker, packaged with Cog for this stuff.[2] Unfortunately Docker doesn't run natively on Mac, so if we want to use the Mac's GPU, we can't use Docker. I wish there was a good container system for Mac. Even better if it were something that spanned both Mac and Linux. (Not as far-fetched as it seems... I used to work at Docker and spent a bit of time looking into this...) [0] https://tvm.apache.org/ [1] https://onnx.ai/ [2] https://github.com/replicate/cog

You can run it with Cog: https://github.com/replicate/cog cog predict r8.im/stability-ai/stable-diffusion -i prompt="cow flying in space" Or you can run the Docker image directly. More details under "run on your own computer" here: https://replicate.com/stability-ai/stable-diffusion

Dokku Maintainer here. Dokku also supports Dockerfiles, Docker Images, Tarballs (similar to heroku slugs), and Cloud Native Buildpacks. I'm also actively working on AWS Lambda support[1] (both for simple usage without much config as well as SAM-based usage) and investigating Replicate's Cog[2] and Railways Nixpacks[3] functionalities for building apps. There are quite a few options in the OSS space (as well as Commercial offerings from new startups and popular incumbents). It's an interesting space to be in, and its always fun to see how new offerings innovate on existing solutions. [1] https://github.com/dokku/lambda-builder [2] https://github.com/replicate/cog [3] https://github.com/railwayapp/nixpacks

Hello HN! One of the creators of Cog here. We built this to deploy models to Replicate (https://replicate.com/), but it can also be used to deploy models to your own infra. Andreas, my co-founder, used to work at Spotify. Spotify wanted to run models inside Docker containers, but Docker was too hard to use for most ML researchers. So, Andreas built a set of templates and scripts to help researchers deploy their own models. This was mixed in with my experience working at Docker. I created Docker Compose, which makes Docker easier to use for dev environments. We were also joined by Zeke, who created Swagger (now OpenAPI), which is used to define a model’s inputs/outputs. Dominic and some other contributors have since joined! https://github.com/replicate/cog#contributors- It’s still early days, so expect a few rough edges, but it’s ready to

Hello HN! Cog lets you package machine learning models in a standard, production-ready container. We built this to deploy models to Replicate[0], where it has been battle hardened with hundreds of models, but it can also be used to deploy models to your own infrastructure. Andreas, my co-founder and one of the creators of Cog, used to work at Spotify. Spotify wanted to run models inside Docker containers, but Docker was too hard to use for most ML researchers. So, Andreas built a set of templates and scripts to help researchers deploy their own models. This was mixed in with my experience working at Docker. I created Docker Compose, which makes Docker easier to use for dev environments. We were also joined by Zeke, who created OpenAPI (née Swagger), which is used to define a model’s inputs/outputs. And, many other contributors have since