File-System Sync

From NAS-Central Buffalo - The Linkstation Wiki
Jump to: navigation, search

File System Synchronization


Rather than a conventional backup I want to recursively replicate complete file-system branches onto my NAS. Then, upon further execution, only new or modified files need to be updated. The initial idea is for a quick and dirty Perl script. Thus it will run for any machine that has Perl installed (e.g. Windows or Linux, etc). Even if the script does not do exactly what you want in its current form, it is easily modifiable.


You'll need a Perl 5 installation to run this script. Other than that there are no requirements (not even a LinkStation).


To run the script, enter the following:
From Windows:

perl [-v][-e exclude_file][-s absolute_source]-d absolute_destination

From Linux: [-v][-e exclude_file][-s absolute_source]-d absolute_destination


-v Specifies give verbose output (all excluded file-system items are listed). 
-e <filename> Specifies the name of a file containing details of files to be excluded from the synchronization. If not specified all files will be synchronized.
-s <path> Specifies the name of the source path from which files will be copied. NOTE: must be an absolute path. If not specified the current directory is used.
-d <path> Specifies the name of the path to which files will be copied. NOTE: must be an absolute path.Files will only be copied if the do not exist or if they have a more recent modification time. Directories will be created as required.

Exclude File

This is a text file where each line specifies a regular expression pattern that, if matched, will exclude the file or directory from being synchronized. Comments may be included by placing a '#' character at the very start of a line.

A sample exclude file is shown below:

# Exclude file for My Photos directory hierarchy for use with
# Exclude files ending with '.log'.
# Exclude some windows files.
# Exclude 'temp' or 'tmp' directories and their contents.

Perl Script

The Perl script is detailed below:

use warnings;
use strict;

use Cwd      qw(abs_path);
use File::Find qw(find);
use File::Copy qw(copy);
use Getopt::Long;

our $verbose;
our @excludes;   # List of exclude patterns.

use Cmd 'abs_path';
our $SRC = abs_path'.';
our $DST;
our $EXCL;
GetOptions("v" => \$verbose,
		 "e=s" => \$EXCL,
		 "s=s" => \$SRC,
		 "d=s" => \$DST);

print "Source <" . $SRC . ">\n";
# A destination is mandatory.
die "Usage: $0 [-v][-e exclude_file][-s source]-d destination" unless defined $DST;

# If an exclude file is define and exists read it.
if (defined $EXCL) {
print "Using exclude list: '$EXCL'.\n";
open(EXCL_FILE, "<$EXCL") or die "Could not open exclude list '$EXCL': $!";
while (<EXCL_FILE>)
  push(@excludes, $_) unless (m/^#/);   # Each line as an exclude.
close EXCL_FILE;

# Start from the specified source.
chdir $SRC or die "Can't chdir to $SRC: $!";

# Ensure that the destination directory exists.
unless (-d $DST) {
mkdir($DST) or die "Can't mkdir $DST: $!";

# Recurse through the file system below the source point.
find { wanted => \&duplicate } => ".";

sub duplicate {
(my $name = $File::Find::name) =~ s!^\./!!;   # Correct name.
return if $name eq ".";
foreach (@excludes) {
  if ($name =~ m/$_/i) {
	print "Excluded: $name with $_\n" if $verbose;
# Get source creation and modification times.
my (undef, undef, undef, undef, undef, undef, undef,
	undef, $src_atime, $src_mtime) = stat("$SRC/$name");
if (-d) {     # Process a directory
  unless (-d "$DST/$name") {
	mkdir("$DST/$name") or die "Can't mkdir $DST/$name: $!";
	utime $src_atime, $src_mtime, "$DST/$name";
	print "Created: $DST/$name\n";
} elsif (-f) { # Process a "normal" file.
  unless (-e "$DST/$name") {
	copy ("$SRC/$name", "$DST/$name")
			   or die "Failed to copy file $SRC/$name to $DST/$name: $!";
	utime $src_atime, $src_mtime, "$DST/$name";
	print "Copied: $SRC/$name\n";
  } else {
	my (undef, undef, undef, undef, undef, undef, undef,
		undef, $dst_atime, $dst_mtime) = stat("$DST/$name");
	if ($src_mtime > (${dst_mtime} + 1)) {
	  copy ("$SRC/$name", "$DST/$name")
					   or die "Failed to copy file $SRC/$name to $DST/$name: $!";
	  utime $src_atime, $src_mtime, "$DST/$name";
	  print "Updated: $SRC/$name\n";