zuul

basis just in order to keep CI green and happy. Rather than having a zero-tolerance CI failure policy, developers should be allowed to break CI temporarily, so long as they fix it in a timely manner (within an hour or two). Per-developer breakage metrics, to the extent they are needed, should not be in terms of breakage counts but instead breakage durations. That is, outside of production, it's fine to break stuff and quickly fix it, so long as you don't leave it broken. The big problems are where domain-siloed developers break a zero-tolerance policy because it was necessary to relax it "temporarily", and things stay broken because the policy stays relaxed and cannot be reinstated without sirens going off. Then restoring the CI policy is blocked for everyone by the one guy who knows the AIX quirks for debootstrap or whatever. Instead, breaking changes should be "allowed" where there is a window to fix the error and still move forward. Only when the window closes without a fix should the breaking change be rolled back (automatically). This line of thinking lends itself to formal, automated policy, but this depends first on judgment and cultural approval. As it happens, the best CI system I'm aware of for truly distributed, multiple project integration is OpenStack's Zuul: https://github.com/openstack-infra/zuul I'm not sure if it accommodates my prescription above, but if not, they probably have a better idea.