implement server backups

[choksi81]

If things crash, we'll have no problem finding the most recent svn snapshot. But it will take months to re-build the wiki content, the Trac install, and all sorts of other essential state like the apache config files.

We need something that can backup the entire server machine into a single image that we can instantiate on another machine if necessary. Weekly backups that take place over the weekend are about the right granularity.

[choksi81]

SystemImager? is one potential software to use for this:
http://wiki.systemimager.org/index.php/Quick_Start_HOWTO
http://www.howtoforge.com/how-to-back-up-an-ubuntu-8.10-system-with-systemimager

[choksi81]

I've added some very basic backups to seattlegeni.cs. There are two scripts now in root's crontab: a nightly one to backup all mysql databases the www user has access to, the other a weekly one to make an archive of /home/geni. The backups are stored in /backups, which is only accessible by root.
For the record:

automysqlbackup.sh: nightly backup of all mysql databases the www user has access to

TODO: change the user/pass in the script to be the mysql root user

0 0 * * * /backups/scripts/automysqlbackup.sh

tar-geni-home-dir.sh: Sunday night tar'ing up of the /home/geni directory

20 0 * * 6 /backups/scripts/tar-geni-home-dir.sh
This doesn't backup anything on seattle.cs, only on seattlegeni.cs. Of course, we still need off-server/off-site backups, too.

[choksi81]

I would recommend looking at rsnapshot ( http://rsnapshot.org/). I personally use it and like it a lot. In addition to providing recovery from a total disaster, you can recover from things as simple as accidentally deleting a configuration file or messing it up. (Or, if a package upgrade goes badly, you can try to rollback.) It is more space efficient than full dumps since it does incremental backup (like Time Machine for OSX).
Just my 2 cents...

[choksi81]

It would be good to have incremental backups, but my first priority is just to get regular simple backups of critical data being done and copied to systems in a somewhat different location, especially as the most critical data can't be reliably backed up with just cp/rsync due to the need for db locking.
At the moment 'offsite' backups will mostly mean the systems in Justin's office get backed up to seattlegeni.cs, and seattlegeni.cs gets backed up to a system in Justin's office.
After critical data is being backed up regularly (e.g. trac, svn, seattlegeni databases), then we can look at rsnapshot. I've used similar rsync-based snapshop backups before and have been happy with them (aside from issues such as high memory usage during the backup when the number of files is very large, but I don't think we would run into that on our systems). rsnapshot looks nicer/better maintained then the ones I've used in the past, such as RIBS.

[choksi81]

I've added a wiki page describing the current state of backups: wiki:Backups?

[choksi81]

priority changed from critical to major
Decreasing priority of this because some backups are currently in place for the most critical data. Other things that we may want to backup that currently aren't backed up include /home and /etc.