WEwenbin·Aug 21, 2018
I'm the author of this blog post and the original developer of this scheduler system.
Glad to see a lot of interesting and insightful comments in this thread!
Some contexts here:
0. Like any piece of software, this scheduler system is not perfect for every company -- legacy code, # of engineers, skillsets of existing employees, engineering culture, tech stack, business,...
1. Every engineer in Nextdoor needs to do on-call rotation -- 50+ engineers back in 2014 (100+ now?), when the scheduler was built. It's important to run a system that all 50+ engineers have confidence to debug on a Friday night if things go south. There are some blackbox-type alternatives of cron, which may be great for a small team of backend experts to operate, but may not be a good fit for 50~100 engineers with very diverse skillsets & backend experience. But why ev
Read the rest
ery engineer needs to be on call? Well, that may deserve a new thread of discussions :)
2. In 2014, there were ~200 production jobs with very different computing characteristics (450+ now). Jobs are owned by different teams and different people with different expertise. Jobs are frequently (every few weeks?) added, removed, and updated. Outages most likely happen during code deployment. It’s important to run a system that works well during deployment (and rollback), e.g., always run the RIGHT version of code for hundreds of jobs.
3. The core scheduler system was pretty tiny, which could be easily understood by any engineer in the company. I remembered it took me a few days (probably less than a week) to finish most of the code. The hard part was productionization, e.g., carefully migrating 200 production jobs to the new system, logging, monitoring/alerting at job level, deployment & rollback process, oncall-related things (documentation/training)… With this simple system, we could easily enumerate various failure situations, so oncall engineers have confidence how to respond.
4. I’m not with Nextdoor any more. But turns out this scheduler system is still working well now: https://github.com/Nextdoor/ndscheduler/issues/33#issuecomme... I think the ROI is pretty good -- a 3-man-month project (from design to run all 200+ jobs on the new system) => running 4 years and counting & easy oncall situation for 50~100+ engineers.
5. Why not Jenkins or other open source alternatives? We investigated a bunch of alternatives. Jenkins is great. Back in 2014, Nextdoor used Jenkins for CI (not sure if it's still true now). We ruled out Jenkins (and Rundeck or the like) for operational reasons, e.g., challenges to integrate with existing code deployment process, operational complexity for 50+ engineers with different backgrounds/expertise... Open source alternatives? Well, in 2014, we couldn't find a good Python-based project. It's important to limit the number of languages/external technologies in the tech stack.