ANandrew_·Apr 20, 2022Submission
Ask HN: DB Migration and Deployment Race Strategy
For the last few months, we've been running our database migrations using Go and Goose [1] within our Github Actions workflows as a step that completes after a successful asset deployment. For the sake of the post, assume that the deployed "thing" is a lambda function. Installing Go and Goose for each deployment run costs us about a minute and a half, to two minutes, depending on Github Actions' mood that day. It's a chunk that I'd like to remove from our deployment runs, as maybe 1 out of every 100 deployments has a database migration in tow. Separating the db migrations and Go/Goose installs into a separate job that runs based on whether or not the migrations directory has changes, dependent on the deployment job (or vise versa), seems like a good way to go. However, that presents some race conditions and there doesn't seem to be much (
Read the rest
that I could find) written on best practices for this scenario. Here are the conditions I've come up with:
1. if a migration is run first, then the current thing running against the db could start erroring
2. if a migration is run first, and the deployment fails, a migrate down needs to be immediately performed, the current thing errors until done
3. if a migration is run last, then the new deployed thing could error for a hot moment while migration is running
4. if a migration is run last and fails, then the new deployed thing could error until the migration is fixed
Presently, we're taking the latter approach in running migrations last. We haven't hit any snags there... yet. But the team is small and I'm trying to plan around eventual expansion. What does HN think?
[1] https://github.com/pressly/goose