This document describes a method for generating automatic rotating "snapshot"-style backups on a Unix-based system, with specific examples drawn from the author's GNU/Linux experience. Snapshot backups are a feature of some high-end industrial file servers; they create the illusion of multiple, full backups per day without the space or processing overhead. All of the snapshots are read-only, and are accessible directly by users as special system directories. It is often possible to store several hours, days, and even weeks' worth of snapshots with slightly more than 2x storage. This method, while not as space-efficient as some of the proprietary technologies (which, using special copy-on-write filesystems, can operate on slightly more than 1x storage), makes use of only standard file utilities and the common rsync program, which is installed by default on most Linux distributions. Properly configured, the method can also protect against hard disk failure, root compromises, or even back up a network of heterogeneous desktops automatically.
- sandip's blog
- Login or register to post comments
Comments
Excellent. Thanks for sharing this informative articles. I found another good articles about linux manual. You can check it out at pdfph.com.
regard, very usefull and thankyou ..
I will be setting up a back-up of a remote web-host via rsync over ssh and creating the snapshot style backup as mentioned above on the local machine.
On the local backup machine:
Generate the private/public key pair for passwordless login via ssh to remote host.
Create an empty password by just hitting enter when prompted for password. Protect the private key, which should only be readable by the owner.
Transfer the public key file to the remote host: The backups are on separately partitioned drive and auto-mounted read-only.
In "/etc/auto.master"
In "/etc/auto.bak"
Restart autofs: Create the rsync backup script to run via cron. The script also rotates out the backups creating a snapshot style backup: Change the mode to be executable on the script. Add the script to crontab for a daily incremental snapshot style backup.On the remote host:
Create script to validate the rsync command used.
Change the mode to be executable on the script. Modify the public key and prepend restricted host and command used, separated by just a comma and no spaces. Append the public key to authorized_keys2 file. Modify "/etc/ssh/sshd_config" to accept forced-commands-only for root ssh login. Restart ssh:
That should do it... the result will be an incremental backup of the last 4 days plus a weekly of the month.
I have since modified the rsync_backup.sh script and made it a bit modular with some amount of error checking:
#!/bin/bash
apshots/config {CLIENT}/conf.txt ${CLIENT}/exclude.txt }.DOMAIN.TLD/home/${CLIENT}/ba k/.snapshots
p; E_BADARGS=65
p; if [ "$NUM_ARGS" -ne "$EXPECTED_ARGS" ] p; then p; &nbs p; &nbs p; echo "Usage: `basename $0` {client}" p; &nbs p; &nbs p; exit $E_BADARGS p; fi
p; if [ -f "${CONFIG_FILE}" ]; then p; &nbs p; &nbs p; # source the config file p; &nbs p; &nbs p; . ${CONFIG_FILE} p; else p; &nbs p; &nbs p; echo "ERROR: ${CONFIG_FILE}" does not exist. p; &nbs p; &nbs p; exit 1 p; fi
p; $MOUNT -o remount,rw,nosuid,noexec,nodev $LPATH p; if (( e = $? )); then p; &nbs p; &nbs p; echo "ERROR: $e, Could not remount in ReadWrite mode. Exiting..." p; &nbs p; &nbs p; exit $e p; fi
p; if [ -f daily.0/.rsync_bak_complete ] p; then
p; &nbs p; &nbs p; if [ "`date +%w`" -eq 0 ] p; &nbs p; &nbs p; then p; &nbs p; &nbs p; &nbs p; [ -d weekly.3 ] && rm -rf weekly.3 p; &nbs p; &nbs p; &nbs p; [ -d weekly.2 ] && mv weekly.2 weekly.3 p; &nbs p; &nbs p; &nbs p; [ -d weekly.1 ] && mv weekly.1 weekly.2 p; &nbs p; &nbs p; &nbs p; [ -d weekly.0 ] && mv weekly.0 weekly.1 p; &nbs p; &nbs p; &nbs p; [ -d daily.3 ] && mv daily.3 weekly.0 p; &nbs p; &nbs p; fi
p; &nbs p; &nbs p; [ -d daily.3 ] && rm -rf daily.3 p; &nbs p; &nbs p; [ -d daily.2 ] && mv daily.2 daily.3 p; &nbs p; &nbs p; [ -d daily.1 ] && mv daily.1 daily.2 p; &nbs p; &nbs p; [ -d daily.0 ] && mv daily.0 daily.1 p; fi
p; if [ ! -d daily.0 ]; then
p; &nbs p; &nbs p; mkdir daily.0
p; &nbs p; &nbs p; # Adjust permissions p; &nbs p; &nbs p; $CHMOD 050 daily.0 p; &nbs p; &nbs p; $CHOWN root:${GID} daily.0
p; fi
p; $NICE -n 12 $RSYNC -azR -e "$SSH -i $KEY -p $PORT" --exclude-from=${EXCLUDE_FILE} --delete --timeout=1800 --link-dest=../daily.1 $RUSER@$RHOST:"$RPATH&quo t; $LPATH/daily.0
p; # error 23 = partial transfer, probably temp files p; # error 24 = file vanished p; if (( e = $? )); then p; &nbs p; &nbs p; if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then p; &nbs p; &nbs p; &nbs p; echo "ERROR: $e, Rsync failed. Exiting..." p; &nbs p; &nbs p; &nbs p; post_sync $e p; &nbs p; &nbs p; fi p; fi
p; # Confirm backup is complete so files are rotated on the backup side p; $RSYNC -a -e "$SSH -i $KEY -p $PORT" --timeout=10 $RUSER@$RHOST:/.rsync_bak_comp lete $LPATH/daily.0
p; if (( e = $? )); then p; &nbs p; &nbs p; if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then p; &nbs p; &nbs p; &nbs p; echo "ERROR: $e, Rsync CONFIRMATION failed." p; &nbs p; &nbs p; &nbs p; echo "Check backup completion. Exiting..." p; &nbs p; &nbs p; &nbs p; post_sync $e p; &nbs p; &nbs p; fi p; fi
p; $UMOUNT $LPATH p; if (( e = $? )); then p; &nbs p; &nbs p; echo "ERROR: $e, Could not unmount ${LPATH}." p; &nbs p; &nbs p; echo "Check your mounts. Exiting..." p; &nbs p; &nbs p; exit $e p; fi
p; $RM -rf .rsync_start p; popd p; sleep 10 p; unmount p; exit $1
p; pushd $LPATH p; sleep 10 p; if [ -f .rsync_start ]; then p; &nbs p; &nbs p; echo "Rsync still running... Exiting!" p; &nbs p; &nbs p; exit 1; p; fi p; mount_write; p; rotate_backups; p; $TOUCH .rsync_start
# do_rsync.sh
# usage: ./do_rsync.sh {client}
RM=/bin/rm
TOUCH=/bin/touch
NICE=/bin/nice
CHOWN=/bin/chown
CHMOD=/bin/chmod
RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
MOUNT=/bin/mount
UMOUNT=/bin/umount
KEY=/root/.ssh/rsync_keys
/>
CLIENT=$1
EXPECTED_ARGS=1
NUM_ARGS=$#
CONFIG_PATH=/root/scripts/sn
CONFIG_FILE=${CONFIG_PATH}/$
EXCLUDE_FILE=${CONFIG_PATH}/
LPATH=/home/virtual/${CLIENT
# Check the number of args
badargs() {
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
}
# Include conf
include_conf() {
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
}
# Mount for writing
mount_write() {
&nbs
&nbs
&nbs
&nbs
&nbs
}
# rotate backups
rotate_backups() {
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
}
# run the rsync
run_rsync() {
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
}
# Unmount
unmount() {
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
}
# Post Sync
post_sync() {
&nbs
&nbs
&nbs
&nbs
&nbs
}
# Pre Sync
pre_sync() {
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
&nbs
}
# Main
badargs;
include_conf;
pre_sync;
run_rsync;
post_sync 0;
There is a configuration and an exclude file, conf.txt and exclude.txt .
"conf.txt" includes the details of the server that need to be backed up.
PORT=22 br />GID=admin138
RUSER=root
RHOST=DOMAIN.TLD
RPATH='/etc /var/lib/mysql /var/www /var/spool/mail /var/spool/squirrelmail'<
"exclude.txt" file has the list of files to be excluded from being backed up.
# exclude patterns (one per line). s/logs/*
#/var/www/vhosts/*/statistic
/var/www/*/log
When the server crashed and can back up, the mountpoints would fail to automount as the filesystem needed manual repair via fsck.
So, also added the below line to "/etc/fstab" to auto fsck and recover journals on boot.
/dev/hdb2-vg00/lv-bak /mnt/lv-bak & nbsp; ext3 noauto,nodev,noexec,nosuid&nbs p; &nbs p; 0 2
This also allowed me to do quick read-write mounts when needed.