aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorintrigeri <intrigeri@boum.org>2008-01-14 20:32:19 +0000
committerintrigeri <intrigeri@boum.org>2008-01-14 20:32:19 +0000
commitef9e8e0dc2359e114412369d12d21e3d7997e123 (patch)
treef60cc04b7edd84437d3943f0090cef07f0878875
parent18f6031a726c4e2e96049c8ca4f1fc3b5dcc7c00 (diff)
downloadbackupninja-ef9e8e0dc2359e114412369d12d21e3d7997e123.tar.gz
backupninja-ef9e8e0dc2359e114412369d12d21e3d7997e123.tar.bz2
support new duplicity versions:
- general cleanup of dup handler - support remove-older-than command - also run "duplicity cleanup" before anything else, for safety reasons - migrated full/incremental backup switch to new syntax ChangeLog: removed trailing space
-rw-r--r--ChangeLog72
-rw-r--r--handlers/dup.in226
2 files changed, 181 insertions, 117 deletions
diff --git a/ChangeLog b/ChangeLog
index 0c7e6fc..9c7770d 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -2,9 +2,17 @@ version 0.9.6 -- unreleased
backupninja changes
handler changes
- rdiff:
+ dup:
+ . General cleanup
+ . Better support for new duplicity (>= 0.4.4) command line syntax:
+ run remove-older-than when $keep is not set to yes (Closes: #458816),
+ and run "duplicity cleanup" before any other duplicity command; both
+ only trigger a warning on failure, since they should not stop backups
+ from being done. Also migrated full/incremental backup switch to the
+ new syntax.
+ rdiff:
. Fixed ignore_version default value missing
- . Add patch from Matthew Palmer to rdiff handler to incorporate sshoptions
+ . Add patch from Matthew Palmer to rdiff handler to incorporate sshoptions
into options via remote-schema not already specified (Closes: #424639)
wget:
. New handler from rhatto designed to incrementally pull content from
@@ -38,7 +46,7 @@ version 0.9.5 -- December 2, 2007
to be correct, also set TLS to be the default over SSL (Closes: Trac#13)
maildir:
. Added an examples file (Closes: Trac#23)
- . Applied patch from Anarcat that fixes the cp/mkdir calls to not use GNU
+ . Applied patch from Anarcat that fixes the cp/mkdir calls to not use GNU
coreutils options, as well as some bashisms (Closes: Trac#24)
. Fix test mode (Closes: Trac#25)
mysql:
@@ -48,8 +56,8 @@ version 0.9.5 -- December 2, 2007
the data from a backup, but still backup the table structure. This is very
useful in cases where tables contain large amounts of cache data. See the
example.mysql for options, thanks Daniel Bonniot (Closes: #408829)
- . Enhance code for selecting databases by asking MySQL not to give us the
- header (-N), to not draw pretty boxes around the output (-B), send the query
+ . Enhance code for selecting databases by asking MySQL not to give us the
+ header (-N), to not draw pretty boxes around the output (-B), send the query
via -e instead of a pipe and ensure MySQL listens to -B. Thanks to
Matthew Palmer (Closes: #452039).
pgsql:
@@ -83,7 +91,7 @@ version 0.9.5 -- December 2, 2007
. Fixed example in example.sys to detail the __star__ in partitionsfile and
note why its necessary (Closes: #409192)
. Force C locale for sfdisk to ensure english words are found in grep
- . Make directory where output is placed configurable, and create the parent dir
+ . Make directory where output is placed configurable, and create the parent dir
if it doesn't exist (Closes: Trac#1)
ninjareport
. Added first draft of method to aggregate reports from many servers into
@@ -94,7 +102,7 @@ version 0.9.5 -- December 2, 2007
fixed 'make install' bug that failed if /etc/backup.d already existed
changed spaces to tabs in Makefile.am
updated redhat spec file (thanks Adam Monsen)
-
+
version 0.9.4 -- October 6th, 2006
backupninja changes
. Fixed bug in toint(), and thus isnow(), which caused it
@@ -103,7 +111,7 @@ version 0.9.4 -- October 6th, 2006
. Add admingroup option to configuration to allow a group that can
read/write configurations (instead of only allowing root). Checks
and complains about group-readable files only when the group differs
- from the one in the configuration file (default is root as before).
+ from the one in the configuration file (default is root as before).
Thanks to Martin Krafft for the patch (Closes: #370396).
. When determining which backup actions to make, find now follows
symlinks for $configdirectory
@@ -117,7 +125,7 @@ version 0.9.4 -- October 6th, 2006
Added rsnap handler:
. rotated rsync snapshops
. code from paulv@bikkel.org
- Added rub handler:
+ Added rub handler:
. alternative to rsnap
. code from rhatto@riseup.net
mysql:
@@ -133,7 +141,7 @@ version 0.9.4 -- October 6th, 2006
. Fixed shell expansion, thanks Thomas Kotzian (Closes: #363297)
. postgres user UID is now the one from inside the vserver if necessary
. Compress now happens in-line to save some disk space (Closes: #370778)
- . $PGSQLUSER is used instead of hardcoding user 'postgres' (although this is the default)
+ . $PGSQLUSER is used instead of hardcoding user 'postgres' (although this is the default)
svn:
. Fixed inversed vsname emptiness check
rdiff:
@@ -176,7 +184,7 @@ version 0.9.4 -- October 6th, 2006
vserver:
. init_vservers: fixed Debian bug #351083 (improper readlink syntax)
. found_vservers: escaped special grep repetition character +
- . forced mktemp to use a template with a name to be more compatible with
+ . forced mktemp to use a template with a name to be more compatible with
different versions of mktemp, thanks anarcat
ninjahelper changes
. Recursively ignore subdirs in /etc/backup.d (Closes: #361102)
@@ -193,7 +201,7 @@ version 0.9.4 -- October 6th, 2006
changed /etc/backup.d permissions to 0770 (for admingroup)
minor documentation fixes
improved RPM build process allowing 'make rpm-package' and 'make
- srpm-package' targets, also fixes permissions on man directories,
+ srpm-package' targets, also fixes permissions on man directories,
cleans up RPM-related files during distclean, and adds default
EDITOR for "autogen.sh -f" if none is set. (thanks Robert Napier)
@@ -242,7 +250,7 @@ version 0.9.3 -- February 1st, 2006
. init_vservers: warn if vservers are enabled but no vserver is found
. new function: vservers_exist
known bugs:
- easydialog:
+ easydialog:
. formDisplay does not return exit status.
version 0.9.2 -- December 29, 2005
@@ -256,7 +264,7 @@ version 0.9.2 -- December 29, 2005
trac:
. mkdir subdirectory problem fixed
duplicity:
- . globbing support fixed in include and exclude options
+ . globbing support fixed in include and exclude options
. different signing and encrypting key support added
. fixed erroneous comments in example.dup about the way
GnuPG-related options are used
@@ -270,12 +278,12 @@ version 0.9.2 -- December 29, 2005
rdiff-backup:
. used to expand '*' in default source directories
. the "Cancel" buttons used to have a weird behaviour
- . updated to include Vserver selection
+ . updated to include Vserver selection
pgsql:
- . forbid the user to choose an empty database set
+ . forbid the user to choose an empty database set
. "Cancel" button now does what it is meant to do
mysql:
- . enhanced for vserver support
+ . enhanced for vserver support
. now able to select databases and dump directory
duplicity:
. new handler added (with Vserver support)
@@ -291,20 +299,20 @@ version 0.9.1 -- November 05 2005
changed subversion handler to use svnadmin hotcopy instead of
the unsupported hot-copy.py script, which was moved in Debian
update rdiff ninjahelper
- now detects and auto-install rdiff-backup on the remote
+ now detects and auto-install rdiff-backup on the remote
machine if possible, also tests the remote backup directory
and offers to create it if it doesn't exist
version 0.9 -- October 19 2005
*** IMPORTANT CHANGE, UPGRADE AT ONCE ***
- fixed insecure temporary file creation
+ fixed insecure temporary file creation
*****************************************
removed erroneous magic file marker in pgsql handler
fixed incorrect find positional
changed direct grep of /etc/passwd to getent passwd.
rdiff helper has much better information on failed ssh attempt
(patch from cmccallum@thecsl.org).
- rdiff handler now supports remote source and local dest.
+ rdiff handler now supports remote source and local dest.
(patch from cmccallum@thecsl.org).
man pages are greatly improved.
@@ -353,7 +361,7 @@ version 0.6 -- June 16 2005
slapcat), restart, passwordfile and binddn. Default backup method
is set to ldapsearch as this is safer
******************************************************************
- NOTE: to get the previous default behavior with the ldap handler,
+ NOTE: to get the previous default behavior with the ldap handler,
you must set "method = slapcat". The new default is ldapsearch.
******************************************************************
implemented fix so that the main script will echo fatal errors rather
@@ -369,10 +377,10 @@ version 0.6 -- June 16 2005
added "testconnect" option
added "sshoptions" option (will be passed to ssh/scp)
added "bandwidthlimit" option
- example.dup example config file for duplicity handler
+ example.dup example config file for duplicity handler
added trac (http://trac.edgewall.com/) environment handler (thanks Charles Lepple!)
added configfile option to mysql handler
- the default is /etc/mysql/debian.cnf. with this,
+ the default is /etc/mysql/debian.cnf. with this,
sqldump doesn't need dbusername. (hotcopy still does).
fixed bug in mysql handler which caused some passwords to not work.
(.my.cnf files now have double quotes around password)
@@ -380,15 +388,15 @@ version 0.6 -- June 16 2005
version 0.5 -- April 12 2005
rdiff handler works when remote sshd has a banner
- rdiff handler supports local dest
+ rdiff handler supports local dest
logfile is created if it doesn't exist
- added "when = hourly"
+ added "when = hourly"
added optional 'nicelevel' to rdiff handler
fixed bug where actions were not run in numeric order.
improved 'when' parsing.
version 0.4.4 -- March 18 2005
- results of handlers are now read line by line.
+ results of handlers are now read line by line.
changes to rdiff handler: added "options", and "keep" is
not necessarily days now (ie, it will pass straight through to
rdiff-backup if the keep has a unit on it, otherwise it adds the 'D').
@@ -396,7 +404,7 @@ version 0.4.4 -- March 18 2005
added maildir handler (very specialized handler)
added --run option (runs the specified action file)
improved sys handler, now uses hwinfo
- added subversion hotbackup handler, svn.
+ added subversion hotbackup handler, svn.
added PATH to cron.d file, which fixes file not found errors.
version 0.4.2 -- Jan 6 2005
@@ -409,7 +417,7 @@ version 0.4.1 -- Jan 3 2005
fixed major bug, 'when' actually works now.
replaced debug function with debug, info, warning, error, fatal.
added --now option to force all actions to be performed now.
-
+
version 0.4 -- Dec 26 2004
added "when" option, so that all configs can specify when
they are to be run.
@@ -417,7 +425,7 @@ version 0.4 -- Dec 26 2004
added .sys handler (hardware, packages, partitions).
version 0.3.4 -- Dec 8 2004
- fixed numerical variable quoting compatibility with older wc
+ fixed numerical variable quoting compatibility with older wc
fixed stderr redirect bug
some comments in example.rdiff
@@ -440,7 +448,7 @@ version 0.3.1 -- Sept 5 2004
force only root can read /etc/backup.d/*
fixed missing equals symbols in example.rdiff
changed backupninja executable to be /usr/sbin rather than /usr/bin
-
+
version 0.3 -- Oct 20 2004
** IMPORTANT ** all config files are now ini style, not apache style
rewrote all scripts in bash for portability
@@ -452,7 +460,7 @@ version 0.2 -- Oct 14 2004
fixed bug: removed printr of excludes (!)
added support for changing the user/group in rdiff sources.
added support for .mysql config files.
-
+
version 0.1 -- Oct 8 2004
- initial release
+ initial release
diff --git a/handlers/dup.in b/handlers/dup.in
index 055531f..00f4b58 100644
--- a/handlers/dup.in
+++ b/handlers/dup.in
@@ -31,11 +31,13 @@ getconf destdir
getconf destuser
destdir=${destdir%/}
-[ "$destdir" != "" ] || fatal "Destination directory not set"
-[ "$include" != "" ] || fatal "No source includes specified"
+### SANITY CHECKS ##############################################################
-### vservers stuff ###
+[ -n "$destdir" ] || fatal "Destination directory not set"
+[ -n "$include" ] || fatal "No source includes specified"
+[ -n "$password" ] || fatal "The password option must be set."
+### VServers
# If vservers are configured, check that the ones listed in $vsnames do exist.
local usevserver=no
if [ $vservers_are_available = yes ]; then
@@ -54,109 +56,132 @@ else
[ -z "$vsinclude" ] || warning 'vservers support disabled in backupninja.conf, vsincludes configuration lines will be ignored'
fi
-
-### see if we can login ###
-
+### See if we can login on $desthost
if [ "$testconnect" == "yes" ]; then
- debug "ssh $sshoptions -o PasswordAuthentication=no $desthost -l $destuser 'echo -n 1'"
- if [ ! $test ]; then
- result=`ssh $sshoptions -o PasswordAuthentication=no $desthost -l $destuser 'echo -n 1'`
- if [ "$result" != "1" ]; then
- fatal "Can't connect to $desthost as $destuser."
- else
- debug "Connected to $desthost as $destuser successfully"
- fi
- fi
+ debug "ssh $sshoptions -o PasswordAuthentication=no $desthost -l $destuser 'echo -n 1'"
+ if [ ! $test ]; then
+ result=`ssh $sshoptions -o PasswordAuthentication=no $desthost -l $destuser 'echo -n 1'`
+ if [ "$result" != "1" ]; then
+ fatal "Can't connect to $desthost as $destuser."
+ else
+ debug "Connected to $desthost as $destuser successfully"
+ fi
+ fi
fi
-### COMMAND-LINE MANGLING ###
+### COMMAND-LINE MANGLING ######################################################
+
+### initialize $execstr*
+execstr_command=
+execstr_options="$options --no-print-statistics"
+execstr_source=
+execstr_serverpart="scp://$destuser@$desthost/$destdir"
+### duplicity version
duplicity_version="`duplicity --version | @AWK@ '{print $2}'`"
duplicity_major="`echo $duplicity_version | @AWK@ -F '.' '{print $1}'`"
duplicity_minor="`echo $duplicity_version | @AWK@ -F '.' '{print $2}'`"
duplicity_sub="`echo $duplicity_version | @AWK@ -F '.' '{print $3}'`"
-# 1. duplicity >= 0.4.2 needs --sftp-command (NB: sftp does not support the -l option)
-# 2. duplicity >= 0.4.3 replaces --ssh-command with --ssh-options, which:
-# - is passed to scp and sftp commands by duplicity
-# - has a special syntax we can not directly feed the command line with
-# (e.g. "IdentityFile=/root/.ssh/id_dsa_backupninja Port=2222", without the -o)
-# so we don't use it: since this version does not use the ssh command anymore,
-# we keep compatibility with our previous config files by passing $sshoptions to
-# --scp-command and --sftp-command ourselves
+### ssh/scp/sftp options
+# 1. duplicity >= 0.4.2 needs --sftp-command
+# (NB: sftp does not support the -l option)
+# 2. duplicity 0.4.3 to 0.4.9 replace --ssh-command with --ssh-options, which is
+# passed to scp and sftp commands by duplicity. We don't use it: since this
+# version does not use the ssh command anymore, we keep compatibility with
+# our previous config files by passing $sshoptions to --scp-command and
+# --sftp-command ourselves
scpoptions="$sshoptions"
[ "$bandwidthlimit" == 0 ] || scpoptions="$scpoptions -l $bandwidthlimit"
-execstr="$options --no-print-statistics "
-
# < 0.4.2 : only uses ssh and scp
if [ "$duplicity_major" -le 0 -a "$duplicity_minor" -le 4 -a "$duplicity_sub" -lt 2 ]; then
- execstr="$execstr --scp-command 'scp $scpoptions' --ssh-command 'ssh $sshoptions' "
+ execstr_options="${execstr_options} --scp-command 'scp $scpoptions' --ssh-command 'ssh $sshoptions'"
# >= 0.4.2 : also uses sftp, --sftp-command option is now supported
else
sftpoptions="$sshoptions"
# == 0.4.2 : uses ssh, scp and sftp
if [ "$duplicity_major" -eq 0 -a "$duplicity_minor" -eq 4 -a "$duplicity_sub" -eq 2 ]; then
- execstr="$execstr --scp-command 'scp $scpoptions' --sftp-command 'sftp $sftpoptions' --ssh-command 'ssh $sshoptions' "
+ execstr_options="${execstr_options} --scp-command 'scp $scpoptions' --sftp-command 'sftp $sftpoptions' --ssh-command 'ssh $sshoptions'"
# >= 0.4.3 : uses only scp and sftp, --ssh-command option is not supported anymore
else
- execstr="$execstr --scp-command 'scp $scpoptions' --sftp-command 'sftp $sftpoptions' "
+ execstr_options="${execstr_options} --scp-command 'scp $scpoptions' --sftp-command 'sftp $sftpoptions'"
fi
fi
-# deal with symmetric or asymmetric (public/private key pair) encryption
+### Symmetric or asymmetric (public/private key pair) encryption
if [ -n "$encryptkey" ]; then
- execstr="${execstr}--encrypt-key $encryptkey "
- debug "Data will be encrypted with the GnuPG key $encryptkey."
+ execstr_options="${execstr_options} --encrypt-key $encryptkey"
+ debug "Data will be encrypted with the GnuPG key $encryptkey."
else
- debug "Data will be encrypted using symmetric encryption."
+ debug "Data will be encrypted using symmetric encryption."
fi
-# deal with data signing
+### Data signing (or not)
if [ "$sign" == yes ]; then
- # duplicity is not able to sign data when using symmetric encryption
- [ -n "$encryptkey" ] || fatal "The encryptkey option must be set when signing."
- # if needed, initialize signkey to a value that is not empty (checked above)
- [ -n "$signkey" ] || signkey="$encryptkey"
- execstr="${execstr}--sign-key $signkey "
- debug "Data will be signed will the GnuPG key $signkey."
+ # duplicity is not able to sign data when using symmetric encryption
+ [ -n "$encryptkey" ] || fatal "The encryptkey option must be set when signing."
+ # if needed, initialize signkey to a value that is not empty (checked above)
+ [ -n "$signkey" ] || signkey="$encryptkey"
+ execstr_options="${execstr_options} --sign-key $signkey"
+ debug "Data will be signed will the GnuPG key $signkey."
else
- debug "Data won't be signed."
+ debug "Data won't be signed."
fi
-# deal with GnuPG passphrase
-[ -n "$password" ] || fatal "The password option must be set."
-
-if [ "$keep" != "yes" ]; then
- if [ "`echo $keep | tr -d 0-9`" == "" ]; then
- keep="${keep}D"
- fi
- execstr="${execstr}--remove-older-than $keep "
+### Incremental or full backup mode
+# If incremental==yes, use the default duplicity behaviour: perform an
+# incremental backup if old signatures can be found, else switch to
+# full backup.
+# If incremental==no, force a full backup anyway.
+if [ "$incremental" == "no" ]; then
+ # before 0.4.4, full was an option and not a command
+ if [ "$duplicity_major" -le 0 -a "$duplicity_minor" -le 4 -a "$duplicity_sub" -lt 4 ]; then
+ execstr_options="${execstr_options} --full"
+ else
+ execstr_command="full"
+ fi
fi
-if [ "$incremental" == "no" ]; then
- execstr="${execstr}--full "
+### Temporary directory
+precmd=
+if [ -n "$tmpdir" ]; then
+ if [ ! -d "$tmpdir" ]; then
+ info "Temporary directory ($tmpdir) does not exist, creating it."
+ mkdir -p "$tmpdir"
+ [ $? -eq 0 ] || fatal "Could not create temporary directory ($tmpdir)."
+ fi
+ info "Using $tmpdir as TMPDIR"
+ precmd="${precmd}TMPDIR=$tmpdir "
fi
-execstr_serverpart="scp://$destuser@$desthost/$destdir"
-execstr_clientpart="/"
+### Cleanup old backup sets (or not)
+if [ "$keep" != "yes" ]; then
+ if [ "`echo $keep | tr -d 0-9`" == "" ]; then
+ keep="${keep}D"
+ fi
+ # before 0.4.4, remove-older-than was an option and not a command
+ if [ "$duplicity_major" -le 0 -a "$duplicity_minor" -le 4 -a "$duplicity_sub" -lt 4 ]; then
+ execstr_options="${execstr_options} --remove-older-than $keep"
+ fi
+fi
-### SOURCE ###
+### Source
set -o noglob
# excludes
for i in $exclude; do
str="${i//__star__/*}"
- execstr="${execstr}--exclude '$str' "
+ execstr_source="${execstr_source} --exclude '$str'"
done
-
-# includes
+
+# includes
for i in $include; do
[ "$i" != "/" ] || fatal "Sorry, you cannot use 'include = /'"
str="${i//__star__/*}"
- execstr="${execstr}--include '$str' "
+ execstr_source="${execstr_source} --include '$str'"
done
# vsincludes
@@ -165,43 +190,74 @@ if [ $usevserver = yes ]; then
for vi in $vsinclude; do
str="${vi//__star__/*}"
str="$VROOTDIR/$vserver$str"
- execstr="${execstr}--include '$str' "
+ execstr_source="${execstr_source} --include '$str'"
done
done
fi
set +o noglob
-### deal with tmpdir ###
-precmd=
-if [ -n "$tmpdir" ]; then
- if [ ! -d "$tmpdir" ]; then
- info "Temporary directory ($tmpdir) does not exist, creating it."
- mkdir -p "$tmpdir"
- [ $? -eq 0 ] || fatal "Could not create temporary directory ($tmpdir)."
+### EXECUTE ####################################################################
+
+execstr_source=${execstr_source//\\*/\\\\\\*}
+
+### Cleanup commands (duplicity >= 0.4.4)
+
+# cleanup
+if [ "$duplicity_major" -ge 0 -a "$duplicity_minor" -ge 4 -a "$duplicity_sub" -ge 4 ]; then
+ debug "$precmd duplicity cleanup $execstr_options $execstr_serverpart"
+ if [ ! $test ]; then
+ export PASSPHRASE=$password
+ output=`nice -n $nicelevel \
+ su -c \
+ "$precmd duplicity cleanup $execstr_options $execstr_serverpart 2>&1"`
+ exit_code=$?
+ if [ $exit_code -eq 0 ]; then
+ debug $output
+ info "Duplicity cleanup finished successfully."
+ else
+ debug $output
+ warning "Duplicity cleanup failed."
+ fi
fi
- info "Using $tmpdir as TMPDIR"
- precmd="${precmd}TMPDIR=$tmpdir "
fi
-### EXECUTE ###
-
-execstr=${execstr//\\*/\\\\\\*}
+# remove-older-than
+if [ "$keep" != "yes" ]; then
+ if [ "$duplicity_major" -ge 0 -a "$duplicity_minor" -ge 4 -a "$duplicity_sub" -ge 4 ]; then
+ debug "$precmd duplicity remove-older-than $keep $execstr_options $execstr_serverpart"
+ if [ ! $test ]; then
+ export PASSPHRASE=$password
+ output=`nice -n $nicelevel \
+ su -c \
+ "$precmd duplicity remove-older-than $keep $execstr_options $execstr_serverpart 2>&1"`
+ exit_code=$?
+ if [ $exit_code -eq 0 ]; then
+ debug $output
+ info "Duplicity remove-older-than finished successfully."
+ else
+ debug $output
+ warning "Duplicity remove-older-than failed."
+ fi
+ fi
+ fi
+fi
-debug "$precmd duplicity $execstr --exclude '**' / $execstr_serverpart"
+### Backup command
+debug "$precmd duplicity $execstr_command $execstr_options $execstr_source --exclude '**' / $execstr_serverpart"
if [ ! $test ]; then
- export PASSPHRASE=$password
- output=`nice -n $nicelevel \
- su -c \
- "$precmd duplicity $execstr --exclude '**' / $execstr_serverpart 2>&1"`
- code=$?
- if [ $code -eq 0 ]; then
- debug $output
- info "Duplicity finished successfully."
- else
- debug $output
- fatal "Duplicity failed."
- fi
-fi
+ export PASSPHRASE=$password
+ output=`nice -n $nicelevel \
+ su -c \
+ "$precmd duplicity $execstr_command $execstr_options $execstr_source --exclude '**' / $execstr_serverpart 2>&1"`
+ exit_code=$?
+ if [ $exit_code -eq 0 ]; then
+ debug $output
+ info "Duplicity finished successfully."
+ else
+ debug $output
+ fatal "Duplicity failed."
+ fi
+fi
return 0