rsync script: Backup all entries listed by rsync

[^rooker]

Here's a small script I wrote for a centralized backup environment.
Example: 4 workstations and 1 server. Each workstation has rsync running as demon, and the server backs up the folder from the clients using rsync.

In order to make this easier and maintenance free, I wrote a small script that queries rsync for a list of folders it is responsible for and synchronizes each "published" entry.

Code: Select all

#!/bin/bash
# @author: ^Rooker
# @date: 13.Aug.2007
# @description: Gets a list of published rsync entries from a client, retrieves all of them and finally compresses each one - or makes a hardlink copy (my favorite).

RSYNC="/usr/bin/rsync"
RSYNC_PORT=873
RSYNC_HOST="to-be-backed-up.mydomain.com"
RSYNC_GET_LIST="$RSYNC --port $RSYNC_PORT $RSYNC_HOST::"

DEST_DIR="/here/is/my/backup/folder"
OLD_DIR=`pwd`
TIMESTAMP=`date +%Y%m%d_%H%M`


function get_backup_list
{
        # use 'sed' to strip the comment of each rsync entry:
        BACKUP_LIST=`$RSYNC_GET_LIST | sed -e 's/\s".*"/ /g'`
}

function pack_service
{
        FILENAME="$TIMESTAMP/$1.tar.bz2";

        echo "Packing $1 as \"$FILENAME\"..."
        mkdir -p $TIMESTAMP
        tar -cjf $FILENAME ./$1
}


function make_snapshot
{
        SNAPDIR=$TIMESTAMP
        mkdir -p $SNAPDIR
        cp -al ./$1 $SNAPDIR/
}


function retrieve_services
{
        for service in $1; do
                echo "Fetching '$service'...   ";
                $RSYNC -azt --progress --numeric-ids --delete --timeout=600 --port $RSYNC_PORT $RSYNC_HOST::$service ./$service

                #pack_service $service
                make_snapshot $service
        done
}

cd $DEST_DIR
get_backup_list
retrieve_services "$BACKUP_LIST"
cd $OLD_DIR

The uncompressed folders in the root of $DEST_DIR are left on purpose to make it easier for rsync to find changes and only transmit deltas.

(Edit on Oct.29th, 2007: Added ability to use hardlink snapshots)

[^rooker]

Brent Norris wrote a great article about installing rsync as a windows-service.
In order to make things more comfortable, I've created an installer (using InnoSetup)

I've offered Brent to put my installer (including its source) on its site, or at least referring to, but he never replied.

That's why I'm posting links to the installer here:
rsync_service_windows-2.6.6.exe

As well as the InnoSetup source to compile/modify it yourself:
rsync_win_installer-2.6.6-src.zip


Unicode?
I've had problems with special characters (e.g. german umlauts) transferred from Windows to Linux. My bash displayed them as question marks (?) and when browsing the files through Samba, they turned to underscores (_). *sigh*

Luckily, the nice guys from OKI Software wrote a UTF-8 version of the cygwin1.dll. I've replaced the DLL coming from Norris' website with the UTF8 version and now I can transfer special characters, too!

Here's the Unicode-aware version of the rsync windows service:
rsync_service_windows-2.6.6_utf8.exe (Installer)
rsync_win_installer-2.6.6_utf8-src.zip (Installer source)

[^rooker]

Here's an excellent article about handling remote/local backups using rsync:
Back up like an expert with rsync

[^rooker]

[PROBLEM]
I wanted to sync 2 folders using:

Code: Select all

rsync -avL --delete /path/to/folder1/* /path/to/target/

Files were copied fine, but files removed from folder1 should be deleted in the target folder, which just didn't happen - except for files in subdirectories.

strange...

[SOLUTION]
Thanks to [url=http://lists.samba.org/archive/rsync/20 ... W.Schulz's answer on someone's post on the rsync mailing list[/url]
it was clear that it was my fault.

It should be:

Code: Select all

rsync -avL --delete /path/to/folder1/ /path/to/target

(no "*" at the source and no "/" at the target)