lhm

Depends on the operation. For most operations there are backwards compatible options that generally result in a 2-step process. Soundcloud (I don't work there) developed something in Ruby for online database migrations that copies the table and uses triggers to handle everything. Note that this only works for MySQL. For PostgreSQL or others, the best option is taking a backwards compatible approach. https://github.com/soundcloud/lhm

This article seems to be about really fundamental changes to your data modeling. If you're using MySQL and interested in only changing one table, i.e. adding/removing a column or an index, check out LHM: https://github.com/soundcloud/lhm I've used it in the past a lot. You have to make sure to throttle the migration, or else it'll max out your CPU & memory.

Apparently gh-ost won't work, but there are other tools that work fine on RDS: I've been using https://github.com/soundcloud/lhm On some fairly large tables (30 million + rows) without any problems

https://github.com/soundcloud/lhm Saved my life a few time :)

Yeah; I tried to stay away from really low-level aspects (just because they're hard to generalize across languages), and also just because the damn thing was so long already, ha. :) As far as database migrations are concerned, GitHub (and others, of course) takes the perspective of migrating before the code that uses those migrations go out. In other words, as @herge says in a sibling comment, the code that gets pushed needs to support two branches of code and two branches of data simultaneously. It's certainly some extra work (and can be pretty gnarly depending on the scope of the migration), but once you get to a certain point it's kind of the only way to do no-downtime migrations. There's many possibilities to help with the actual migration process, depending on what database you're using. With MySQL, for example, you can do something

For ActiveRecord (or other Ruby-based) migrations with MySQL, I've had good luck with Large Hadron Migrator[1]. It's worked flawlessly for some of our own tables with several hundred million rows (and no downtime). [1] https://github.com/soundcloud/lhm

> How would you then ever modify an existing database? You would use techniques like: https://github.com/soundcloud/large-hadron-migrator """ That is until your tables grow to millions of records. At this point, the locking nature of ALTER TABLE may take your site down for an hour or more while critical tables are migrated. In order to avoid this, developers begin to design around the problem by introducing join tables or moving the data into another layer. Development gets less and less agile as tables grow and grow. To make the problem worse, adding or changing indices to optimize data access becomes just as difficult. Side effects may include black holes and universe implosion. There are few things that can be done at the server or engine level. It is possible to change default values in an ALTER TABLE without locking the table. The In

I've been eyeing Soundcloud's Large Hadron Migrator (https://github.com/soundcloud/large-hadron-migrator). I would love to see a Django/South specific implementation.

We use https://github.com/soundcloud/large-hadron-migrator/

Data migrations are done carefully with the Large Hadron Migrator Ruby gem: https://github.com/soundcloud/large-hadron-migrator. Facebook has a similar tool: https://www.facebook.com/note.php?note_id=430801045932

At SoundCloud we developed Large Hadron Migrator [1] to work around the problem of locked tables with ALTER and MySQL. It creates a table with the new schema, copies the data from the "old" table to the new one in chunks (utilizing INSERT IGNORE and triggers) and then switches the two tables. The switching is not yet atomic, as there was a bug with RENAME TABLE. [1] https://github.com/soundcloud/large-hadron-migrator Edit: URL