cortex

Hi, I am searching for benchmarks that compare the performance of various machine learning model serving frameworks. Some of the previous posts such as the following exist but they don't paint the full picture. Is there any reliable benchmark that gives a good snapshot of the state of art? 1. https://news.ycombinator.com/item?id=28760158 2. https://github.com/cortexlabs/cortex/tree/v0.15.1

The price of GPU inference can be brutal, but there's a lot you can do on the infra side to improve it: - Spot instances - Aggressive autoscaling - Micro batching Can reduce inference compute spend by huge amounts (90% is not uncommon). ML, especially anything involving realtime inference, is an area where effective platform engineering makes a ridiculous difference even in the earliest days. Source: I help maintain open source ML infra for GPU inference and think about compute spend way too much https://github.com/cortexlabs/cortex

I was expecting this to be more about running inference in production, though the information in the article itself was interesting on its own. There does seem to be a dearth of writing on the actual topic of deploying models as prediction APIs, however. I work on an open source ML deployment platform ( https://github.com/cortexlabs/cortex ) and the problems we spend the most time on/teams struggle with the most don't seem to be written about very often, at least in depth (e.g. How do you optimize inference costs? When should you use batch vs realtime? How do you integrate retraining, validation, and deployment into a CI/CD pipeline for your ML service?). Not taking anyway from the article of course, it is well written and interesting imo.