cog

Cloudflare | Systems or ML Engineer, Workers AI | Austin, TX or London, UK or San Francisco, CA (Hybrid) | Full-Time | https://cloudflare.com Cloudflare is building across the entire AI stack. Here are some exciting things that we have launched recently: - Kimi k2.5: https://blog.cloudflare.com/workers-ai-large-models/ - Kimi Performance Improvements: https://blog.cloudflare.com/high-performance-llms/ - Kimi k2.6 Speculative Decoding and Shared KV Cache: https://x.com/kevin_flansburg/status/2050238819065299110 - Unweight Tensor Compression: https://blog.cloudflare.com/unweight-tensor-compression/ - Code Mode MCP Servers: https://blog.cloudflare.com/code-mode/ - Agents SDK: https://blog.cloudflare.com/building-agents-with-openai-and-... - Agent Memory: https://blog.cloudflare.com/introducing-agent-memory/ - Internal AI Tooling: https://blo

Cloudflare | Systems or ML Engineer, Workers AI | Austin, TX or London, UK or San Francisco, CA (Hybrid) | Full-Time | https://cloudflare.com Cloudflare is building across the entire AI stack. Here are some exciting things that we have launched recently: - Release Cog v0.17: https://github.com/replicate/cog/releases/tag/v0.17.0 - Dynamic Worker Sandboxes: https://blog.cloudflare.com/dynamic-workers/ - Kimi K2.5: https://blog.cloudflare.com/workers-ai-large-models/ - Code Mode MCP Servers: https://blog.cloudflare.com/code-mode/ - Agents SDK: https://blog.cloudflare.com/building-agents-with-openai-and-... We are looking for systems and ML engineers to help build our edge inference platform: - Senior Systems Engineer, Workers AI - https://job-boards.greenhouse.io/cloudflare/jobs/7764827?gh_... - Senior / Principal Machine Learning Engineer,

I wonder if they'll continue to support and develop Cog [0], which I've found quite useful for local deployments of models (as well as to Replicate's servers). [0] https://github.com/replicate/cog

I knew about Replicate but not about Cog, so linky if others are similarly interested https://github.com/replicate/cog#how-it-works (Apache 2)

Can fly run cog files like replicate uses? Would be nice to take those pre packaged models run them here with the same prediction API Maybe cos it's replicate they might be hesitant to adopt it but it does seem to make things a lot smoother Even with lambalabs' lambdastack I still hit cuda hell https://github.com/replicate/cog

Founder of Replicate here. We open pull requests on models[0] to get them running on Replicate so people can try out a demo of the model and run them with an API. They're also packaged with Cog[1] so you can run them as a Docker image. Somebody happened to stumble across our fork of the model and submitted it. We didn't submit it nor intend for it to be an ad. I hope the submission gets replaced with the upstream repo so the author gets full credit. :) [0] https://github.com/Vaibhavs10/insanely-fast-whisper/pull/42 [1] https://github.com/replicate/cog

I'm in the same situation. I found this cog project to dockerise ML https://github.com/replicate/cog : you write just one python class and a yaml file, and it takes care of the "CUDA hell" and deps. It even creates a flask app in front of your model. That helps keep your system clean, but someone with big $s please rewrite pytorch to golang or rust or even nodejs / typescript.

the cog template is just starter code to make it super simple to deploy llama-v2 on any infrastructure of your choosing! More about cog https://github.com/replicate/cog Our thinking was just that a bunch of folks will want to fine-tune right away, then deploy the fine-tunes, so trying to make that easy... Or even just deploy the models-as-is on their own infra without dealing with CUDA insanity!

They have a docker image, I would start with that. It's pretty easy to setup a container in the cloud these days. They even have a tool to customize the container: https://github.com/replicate/cog

As mentioned in sibling comments, Torch is indeed the glue in this implementation. Other glues are TVM[0] and ONNX[1] These just cover the neural net though, and there is lots of surrounding code and pre-/post-processing that isn't covered by these systems. For models on Replicate, we use Docker, packaged with Cog for this stuff.[2] Unfortunately Docker doesn't run natively on Mac, so if we want to use the Mac's GPU, we can't use Docker. I wish there was a good container system for Mac. Even better if it were something that spanned both Mac and Linux. (Not as far-fetched as it seems... I used to work at Docker and spent a bit of time looking into this...) [0] https://tvm.apache.org/ [1] https://onnx.ai/ [2] https://github.com/replicate/cog

You can run it with Cog: https://github.com/replicate/cog cog predict r8.im/stability-ai/stable-diffusion -i prompt="cow flying in space" Or you can run the Docker image directly. More details under "run on your own computer" here: https://replicate.com/stability-ai/stable-diffusion

Dokku Maintainer here. Dokku also supports Dockerfiles, Docker Images, Tarballs (similar to heroku slugs), and Cloud Native Buildpacks. I'm also actively working on AWS Lambda support[1] (both for simple usage without much config as well as SAM-based usage) and investigating Replicate's Cog[2] and Railways Nixpacks[3] functionalities for building apps. There are quite a few options in the OSS space (as well as Commercial offerings from new startups and popular incumbents). It's an interesting space to be in, and its always fun to see how new offerings innovate on existing solutions. [1] https://github.com/dokku/lambda-builder [2] https://github.com/replicate/cog [3] https://github.com/railwayapp/nixpacks

Hello HN! One of the creators of Cog here. We built this to deploy models to Replicate (https://replicate.com/), but it can also be used to deploy models to your own infra. Andreas, my co-founder, used to work at Spotify. Spotify wanted to run models inside Docker containers, but Docker was too hard to use for most ML researchers. So, Andreas built a set of templates and scripts to help researchers deploy their own models. This was mixed in with my experience working at Docker. I created Docker Compose, which makes Docker easier to use for dev environments. We were also joined by Zeke, who created Swagger (now OpenAPI), which is used to define a model’s inputs/outputs. Dominic and some other contributors have since joined! https://github.com/replicate/cog#contributors- It’s still early days, so expect a few rough edges, but it’s ready to

Hello HN! Cog lets you package machine learning models in a standard, production-ready container. We built this to deploy models to Replicate[0], where it has been battle hardened with hundreds of models, but it can also be used to deploy models to your own infrastructure. Andreas, my co-founder and one of the creators of Cog, used to work at Spotify. Spotify wanted to run models inside Docker containers, but Docker was too hard to use for most ML researchers. So, Andreas built a set of templates and scripts to help researchers deploy their own models. This was mixed in with my experience working at Docker. I created Docker Compose, which makes Docker easier to use for dev environments. We were also joined by Zeke, who created OpenAPI (née Swagger), which is used to define a model’s inputs/outputs. And, many other contributors have since