dotmesh

model makes a bad decision in production, you need to know what dataset it was trained on, which means needing to be able to retrieve that dataset. That's because you can't isolate and fix the problem without being able to re-run it. Yes, data changes and marches forwards, and that's why you should retrain models. But you also need to be able to go backwards to do robust ML. My 2 cents. I also wrote a DevOps for ML Manifesto: https://dotscience.com/manifesto/ To summarize: 1. All models must be reproducible by someone else 6 months later. 2. All models must be accountable, that means you must be able to justify the basis on which they made their decision, in particular you need to know which data version was used and where it came from (provenance). 3. Model development must be collaborative. That means I need to be able fork a copy of your project and maintain all the metadata about which runs you did, with their respective provenance history. 4. Models must have a continuous lifecycle. You're not done when you ship - because models are about finding patterns in data, and the world is constantly changing, you need statistical monitoring and retraining to compensate for model drift. Do you disagree? Did I miss anything? Let me know your thoughts!