wal-e

Note to all the postgres users cursing EBS right now: you can persist your db instances to s3 with https://github.com/heroku/WAL-E, which, combined with streaming replication and some redundancy, should let you run off ephemeral disk.

I've never used Barman and would love to hear more from people who have used it. I've used WAL-E[1] in production for almost a year now. It was painless to setup and configure and has been working flawlessly for the entire time. I've had to do database recoveries, point-in-time recoveries, and database upgrades, and WAL-E has worked through all of them. [1]: https://github.com/heroku/WAL-E

While looking beyond a pg_dump-style approach to backup/recovery, I was considering https://github.com/heroku/WAL-E and discovered Barman. It's also open source, and looks like a strong contender.

Speaking of backups, you might want to run wal-e: https://github.com/heroku/wal-e It is not the best program I ever principally authored (since I know you are very discriminating Python programmer), however, it does work very well and has an excellent reliability record so far, seemingly for both Heroku and users of wal-e per most reports. I also tried quite hard -- and in this I'm pretty satisfied given both the feedback I've gotten and my personal experiences -- to make the program easy to use and administer, for one server and operator to many servers and one operator. If someone is not doing continuous archiving and can't use our service for any reason, we do try to urge them to use something like wal-e or any of the other continuous archiving options. And to that end, I tried to make a pretty credible wal-e set-up only take a few m

Thankfully the Heroku guys have open sourced WAL-E which manages all the tedious parts of setting this up and keeping it running. https://github.com/heroku/WAL-E

* Check out wal-e (https://github.com/heroku/wal-e), we wrote it and we manage 250k postgres databases with it. :) * libpq does basically suck, but go has a pretty great native implementation. suggestions are welcome for improvement, patches are even better. * pg_cancel_backend() plus select-fu on the `pg_locks` table joined with `pg_stat_activity` = joy * fast count(*) is possible via index-only scans in 9.2

We have the same setup, use local ephemeral disks on EC2 with Postgres. We never even tried EBS as we just heard too much negative things about it, namely its variance in performance. So our approach is to RAID-10 (4) local volumes together. We then use replication to at least 3 slaves, all of which are configured the same and can become master in the event of a failover. We use WAL-E[0] to ship WAL logs to S3. WAL-E is totally awesome. Love it! [0] https://github.com/heroku/wal-e

Another possibility on the AWS cloud is to put the logs on S3, Amazon S3 has High Durability. Heroku has published a tool for doing it: https://github.com/heroku/WAL-E , which they use to manage their database product.

You're exactly right about log streaming to S3. We open sourced the tool that we use to perform that: https://github.com/heroku/wal-e

Postgresql has PITR backup. http://www.postgresql.org/docs/9.1/static/continuous-archivi... This allows you to go back in time to an earlier snapshot of your data, without having to do a full database backup. Unfortunately, it's still a pain to setup, even with tools like https://github.com/greg2ndQuadrant/repmgr. Wish there was more convention over configuration. (I use Heroku's WAL-E to backup the data to S3. https://github.com/heroku/WAL-E)

Not sure why it was dead-ed (possibly a double-post), but here's the answer from an author of it in case you don't have showdead on: fdr 1 hour ago | link | parent [dead] | on: Heroku's AWS outage post-mortem The mechanism is PostgreSQL continuous archiving. http://github.com/heroku/wal-e This tool is still quite nascent. It received quite a trial by fire, having not (before this point) been revealed as a value-added feature to the service just yet in a wide scale.

Hey Jedberg, if you guys aren't already rolling your own, check out fdr's WAL-E tool. It bounces postgres write-ahead logs off S3 and goes great with the new PG9 replication. https://github.com/heroku/WAL-E