Building a secure network backup server

I've been taking digital photos for a few years and now have a collection of photos and videos with significant value to me. Most of these photos only exist on the hard disk in my computer. Since hard disks do occasionally break, the only way I can be sure my photos will be preserved is to keep backup copies of the files.

In this post I'll describe the backup system I built. It automatically backs up each computer in my house and keeps a secure copy at my parent's house.

(Thanks to MISS J for this illustration)

There were a number of design goals for this system.

1. It had to be fully automatic during daily use.
   
2. It needed to backup multiple computers with hundreds of gigabytes of disk storage.
   
3. It needed to only store changes between each backup, so that multiple copies would not consume excessive hard disk space.
4. It must only send the changes across the network connections, since there's not enough time or bandwidth to send everything every day.
   
5. It needed to automatically maintain an off-site copy.
6. The off-site copy needed to be encrypted before it was sent over the network so there would be no expectation of security or privacy on the internet or at the off-site location.
   

An overview of how it works:

Linux and Mac computers run backup scripts that perform these tasks:

1. turn on the backup server
2. rotate a set of backups
3. use rsync + ssh + encfs to make a new backup
4. tell the backup server they are finished so it can turn itself off

The backup server is a computer with a big hard disk and no monitor. It runs Debian because it's very stable and easy to install without the graphic interface. The OS is just the basic Debian installation plus a few extra packages. If the off-site backup is more than 1 day old the server updates it before turning itself off.

The off-site location can be any computer that you have ssh access to. In my case there was an OpenWRT router available that I attached a USB hard disk to and added my public ssh key to allow automated connections.


Setting up the backup server:

1. install Debian on an old computer with a big hard drive.
   
   • I am using a AMD 1.4Ghz with a 500GB SATA drive
   • desktop partitioning, 4GB for / and the rest for /home
   • don't install any extra packages from the installer
     
   • apt-get install ntpdate
     
   
   
2. install extra packages
   
   • apt-get install ssh
   • edit /etc/ssh/sshd_config
     PermitRootLogin no
   
   
3. enable the serial console (optional)
   
   • edit /boot/grub/menu.lst
     ## display the grub menu on the serial port COM1 or the normal console
serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console
# kopt=root=/dev/hda1 ro console=ttyS0,9600 console=tty0

   • update-grub
     
   • edit /etc/inittab to enable login on serial port
     T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100
     
   
   
4. enable wakeonlan
   
   • make sure the PC BIOS will wake from PCI events or Wake-on-Lan signals
     
   • apt-get install ethtool apmd
     
   • add the following line to /etc/network/interfaces
     up ethtool -s eth0 wol g
     
   • reboot then test sending a wakeonlan command
     
   
   
5. install fuse/encfs
   
   • apt-get install module-assistant build-essential
     
   • module-assistant prepare
     
   • module-assistant auto-install fuse
     
   • apt-get install encfs
     
   • modprobe fuse
     
   • add "fuse" to /etc/modules
     
   
   
6. setup sudo permissions in /etc/sudoers so all backup users can touch files in /var/run
   
   User_Alias USERS = user1, user2
Cmnd_Alias SHUTDOWN = /usr/bin/touch /var/run/*
root ALL=(ALL) ALL
USERS ALL = NOPASSWD: SHUTDOWN

   
   
7. setup shutdown_when_idle script
   
   • add shutdown_when_idle script to /etc/crontab to run every minute
     
   • add offsite-backup-sync.sh to /root
   • add delayed-kill to /root
     
   
   

Setup the backup script on each client

Mac users
[UPDATE 3/18/2009 rsync on a Mac does not preserve the resource fork of files like fonts so I'm no longer using this. I'm trying CrashPlan now.]

1. put the backup script in /Applications/Utilities/backup-script.sh
2. edit the backup script so that it uses the appropriate settings for you
   
3. edit the backup_excludes.txt file and add any files you do not want to backup
4. copy your ssh public key to the ~/.ssh/authorized_keys file on the backup server
   
5. put the backup.plist file /Library/LaunchDaemons/backup.plist
   • this will run the backup script every time your network connections change or every few hours
   • the backup script checks that it is connected to your home network and does the backup if it has been more than 20 hours since the last one

Linux users

1. put the backup script in your home directory
   • I put mine in ~/backups/backup-script.sh
2. edit the backup script so that it uses the appropriate settings
3. edit the backup_excludes.txt file and add any files you do not want to backup
4. copy your ssh public key to the ~/.ssh/authorized_keys file on the backup server
5. add a cron job to run the backup script every day or as frequently as you want

Always room for improvement:
An encryption method that is rsync friendly and could hide the directory structure would make the system more secure. One other system that looks like it may also fulfill my requirements is Areca Backup, and I may give it more consideration in the future.