Contents

Logarithmic backups on a linux server

Contents

This is the second part of the lab server setup guide. Mainly for my own memory purposes, but if you find it useful then so much the better.

Synchronization

As previously discussed all of the machines in the lab are mounted onto the server using NFS under /media/lab-machine-XX. But this does not back them up. So the first step of the backup is to copy the files from the lab machines to the server. For this I use unison which is an easy implementation of rsync which will compare the two versions in question, determine the newest and only copy across changes saving bandwidth and time.

unison /media/lab-machine-01 /media/raid/lab-machine-01

Will copy any changes from the NFS mount /media/lab-machine-01 to the local RAID array.

The backup script

For archiving purposes we don’t want to have raw folders and files all over the place we want to compress them for storage for this, I wrote the following bash script

#! /bin/bash

# Morgan's backup script - use at your peril

# SETUP VARIABLES
location=/media/raid

# Folders to backup
f1=$location/PC1
f2=$location/PC2
f3=$location/PC3
f4=$location/PC4
f5=$location/PC5
f6=$location/PC6
f7=$location/old_users

backuplocation=/media/raid/backup_archive
log=/media/raid/backup_archive/backup.log

# END OF VARIABLES

# Start backup
echo Backup started: `date` >> $log

if [ -d $backuplocation ]; then

tar -cvvf $backuplocation/`date +%Y-%m-%d`.tar $f1 $f2 $f3 $f4 $f5 $f6 $f7

echo Backup completed: `date` >> $log
echo — Backup completed –;

# Otherwise write that backup failed to log file
else
echo FAILED: `date` >> $log
echo – WARNING: –
echo – BACKUP FAILED –;
fi

And saved that as wrDaily (write daily) at /media/raid/backup_archive/backup_scripts. So this script every time it is run will take the folders listed in the variables section and compress them into a TAR folder at the backup location with the name YYYY-MM-DD.tar. A backup log is maintained at /media/raid/backup_archive/backup.log which will record the backup start and end times.

​# End of week clear up If we run the wrDaily script every day then we shall quickly fill up our server, in a year we’d have 365 copies, which would probably fill the server. As a result, on a Monday night we need a script that will deleted the last week’s backups. But for good backup reasons we want to leave the backup from this Monday and the previous Monday well alone. This gives us the second script ​rmWeek

#! /bin/bash


# Morgan's end of week removal. To be run on Monday nights.
#
# This scipt on a Monday night will look at the backups from
# last week. If the backup from last Tuesday was not the 1st
# of the month then it is deleted, and so on for the rest of
# the week up to Sunday night.
#
# Monday night is always left. This is dealt with later in the
# rmMonth script.

# Setup variables
location=/media/raid/backup_archive
log=/media/raid/backup_archive/backup.log

# END OF VARIABLES

# Set dates
today=$(date --date='today' +%Y-%m-%d)
Monday=$(date --date='Monday' +%Y-%m-%d)
lMonday=$(date --date='last Monday' +%Y-%m-%d)

# Start removal, start with a check to see that it is Monday
if [ "$Monday" != "$today" ]; then

echo Cleaning backup archive [weekly]: `date` >> $log

# Get variables for last Tuesday
lY=$(date --date='last Tuesday' +%Y)
lm=$(date --date='last Tuesday' +%m)
ld=$(date --date='last Tuesday' +%d)
lD=$(date --date='last Tuesday' +%Y-%m-%d)

# If last Tuesday was not the 1st of the month then delete it
if [ $lD != $lY-$lm-01 ] ; then
rm -rf $location/$lD.tar
echo Removed $lD.tar >> $log
fi

lY=$(date --date='last Wednesday' +%Y)
lm=$(date --date='last Wednesday' +%m)
ld=$(date --date='last Wednesday' +%d)
lD=$(date --date='last Wednesday' +%Y-%m-%d)

if [ $lD != $lY-$lm-01 ] ; then
rm -rf $location/$lD.tar
echo Removed $lD.tar >> $log
fi

lY=$(date --date='last Thursday' +%Y)
lm=$(date --date='last Thursday' +%m)
ld=$(date --date='last Thursday' +%d)
lD=$(date --date='last Thursday' +%Y-%m-%d)

if [ $lD != $lY-$lm-01 ] ; then
rm -rf $location/$lD.tar
echo Removed $lD.tar >> $log
fi

lY=$(date --date='last Friday' +%Y)
lm=$(date --date='last Friday' +%m)
ld=$(date --date='last Friday' +%d)
lD=$(date --date='last Friday' +%Y-%m-%d)

if [ $lD != $lY-$lm-01 ] ; then
rm -rf $location/$lD.tar
echo Removed $lD.tar >> $log
fi

lY=$(date --date='last Saturday' +%Y)
lm=$(date --date='last Saturday' +%m)
ld=$(date --date='last Saturday' +%d)
lD=$(date --date='last Saturday' +%Y-%m-%d)

if [ $lD != $lY-$lm-01 ] ; then
rm -rf $location/$lD.tar
echo Removed $lD.tar >> $log
fi

lY=$(date --date='last Sunday' +%Y)
lm=$(date --date='last Sunday' +%m)
ld=$(date --date='last Sunday' +%d)
lD=$(date --date='last Sunday' +%Y-%m-%d)

if [ $lD != $lY-$lm-01 ] ; then
rm -rf $location/$lD.tar
echo Removed $lD.tar >> $log
fi

echo Archive clean [weekly]: COMPLETE `date` >> $log

else
echo This script can only be called on a Monday
fi

Note that this script will write to the same backup log file. And that this script has been complicated by not wanting to delete the backup from the 1st of the month (explained in next section)

End of month clean up

At the end of the month we will have (depending on the month) 3-5 Monday night backups. But again, if we left this for a year we’d have 52 copies, so on the 2nd of every month we run a cleaning script to remove any backups from the month-before-last that were not the first of the month.

For example on the 2nd August we want to remove all backups in June, except 1st June, leaving us with 1st June, 1st July and then the Monday night backups for July.

This gives us our third script rmMonth

#! /bin/bash


# Morgan's end of month removal. To be run on 2nd of the month

# Setup variables
location=/media/raid/backup_archive
log=/media/raid/backup_archive/backup.log

# END OF VARIABLES

# Set dates
day=$(date --date='today' +%d)
llyear=$(date --date='2 months ago' +%Y)
llmonth=$(date --date='2 months ago' +%m)


# Start removal, start with a check to see that it is 2nd

if [ "$day" = 2 ]; then

echo Cleaning backup archive [monthly]: `date` >> $log

# Remove 2 to 9
rm -rf $location/$llyear-$llmonth-0[!1].tar
# Remove 10 to 39
rm -rf $location/$llyear-$llmonth-[1-3][0-9].tar

echo Archive clean [monthly]: COMPLETE `date` >> $log

fi

Again, this script uses the same backup log file

End of year clean up

On the 2nd January we want to have a clean up of the archive again to minimize space usage of old backups. For this reason we delete the interim months for the year-before-last leaving the 1st January backup from the year-before-last and then last year will still have it’s 12x 1st of the month backups. This gives us ​rmYear

#! /bin/bash

# Morgan's end of year removal. To be run on 2nd January
#
# This will remove the 1st of the month files from the year before
# last. ie on 2nd January 2013 this will will delete Feb-Dec 2011

# Setup variables
location=/media/raid/backup_archive
log=/media/raid/backup_archive/backup.log

# END OF VARIABLES

# Set dates
llyear=$(date --date='2 years ago' +%Y)


# Start removal, start with a check to see that it is 2nd

if [ "$day" != 2 ]; then

echo Cleaning backup archive [annual]: `date` >> $log

# Remove Feb 02 - Sept 09
rm -rf $location/$llyear-0[2-9]-01.tar
# Remove Oct, Nov, Dec
rm -rf $location/$llyear-1[0-2]-01.tar

# Update log
echo Removed $llyear-02-01.tar >> $log
echo Removed $llyear-03-01.tar >> $log
echo Removed $llyear-04-01.tar >> $log
echo Removed $llyear-05-01.tar >> $log
echo Removed $llyear-06-01.tar >> $log
echo Removed $llyear-07-01.tar >> $log
echo Removed $llyear-08-01.tar >> $log
echo Removed $llyear-09-01.tar >> $log
echo Removed $llyear-10-01.tar >> $log
echo Removed $llyear-11-01.tar >> $log
echo Removed $llyear-12-01.tar >> $log

echo Archive clean [annual]: COMPLETE `date` >> $log

fi

Automation

Finally to automate the backup script we need to tell the server to run these scripts at the correct times. We do this using the computer’s inbuilt task scheduler known as Cron or Crontab. Every minute the computer checks the files inside /etc/cron.d/ and will execute the commands in it. We therefore need to create a file in this folder (requires sudo permissions) and tell it what to do.

Cron jobs require a special format, that I’ll try to explain here:

 *     *     *   *    *        *     command to be executed
 -     -     -   -    -        -
 |     |     |   |    |        |
 |     |     |   |    |        +---- user to execute (eg root)
 |     |     |   |    +----- day of week (0 - 6) (Sunday=0)
 |     |     |   +------- month (1 - 12)
 |     |     +--------- day of month (1 - 31)
 |     +----------- hour (0 - 23)
 +------------- min (0 - 59)

So here is a copy of BACKUPSCRIPT that I created. Remember unison is the file syncing command

# EPR-BACKUP SERVER SCRIPT
#
# This script will:
# Make a backup every night. On a Monday night it will make a
# backup and delete the previous week's backups. The Monday
# before remains untouched
#
# On the first of every month a full backup is made.
#
# On the 2nd of every month the Monday backups from the month
# before are deleted. ie on the 2nd May, April's Monday night
# backups will be deleted but this is ok because we will have
# 1st April and 1st May backups at this point.
#
# On the 2nd January every year, the year-before-last's monthly
# backups are removed. ie in 2013 the 1st Feb-Dec 2011 backups
# are deleted. 1st January 2011 and 1st January 2012 will remain

# Daily backup
00 10 *  *  *  root unison /media/PC1 /media/raid/PC1
02 10 *  *  *  root unison /media/PC2 /media/raid/PC2
04 10 *  *  *  root unison /media/PC3 /media/raid/PC3
06 10 *  *  *  root unison /media/PC4 /media/raid/PC4
08 10 *  *  *  root unison /media/PC5 /media/raid/PC5

00 11 *  *  *  root sh /media/raid/backup_archive/wrDaily

# Monday night, removal of previous week
10 11 *  *  1  root sh /media/raid/backup_archive/rmWeek

# 1st of month file generation
15 11 01 *  *  root sh /media/raid/backup_archive/wrMonth

# 2nd of month deletion of the month-before-last (leaving the 1st)
15 11 02 *  *  root sh /media/raid/backup_archive/rmMonth

# 2nd January clearing of year-before-last's months leaving 1/1
20 11 02 01 *  root sh /media/raid/backup_archive/rmYear

Backup log

Finally for those that are interested, here’s an example of what the resulting backup log would look like

Backup started: Tue Jul 31 15:58:22 BST 2012
Backup completed: Tue Jul 31 16:12:04 BST 2012
Cleaning backup archive [weekly]: Tue Jul 31 16:40:36 BST 2012
Removed 2012-07-24.tar
Removed 2012-07-25.tar
Removed 2012-07-26.tar
Removed 2012-07-27.tar
Removed 2012-07-28.tar
Removed 2012-07-29.tar
Archive clean [weekly]: COMPLETE Tue Jul 31 16:40:36 BST 2012

This page previously appeared on morganbye.net[^1][^2][^3]

[^1:] http://morganbye.net/logarithmic-backups-on-a-linux-server [^2:] http://morganbye.net/2012/08/logarithmic-backups-on-a-linux-server) [^3:] http://morganbye.net/uncategorized/2012/08/logarithmic-backups-on-a-linux-server/