Duplicity vs rdiff-backup in action
Rdiff-backup is one of my favorite tools for backing up some local stuff. Why? Its pretty simple and store full backup 'as is'. So, backup files (main) can be easily accessed.
The main disadvantage I sow - speed. Rdiff-backup may be very slow on some files.
Duplicity is the solution?
Duplicity, as I see, is the next generation of rdiff-backup. It has some useful things, such as: encryption, many destination protocols...and much more!
Let's make a few tests now.
I'll use Docker for creating several isolated containers. Some of them will be a storage and other will be hosts with important data wich we need to backing up.
Feeling lazy? Jump to results and see who is fastest end efficient
Containers in this example:
data01 -- local node with data for rdiff-backup
data02 -- remote node with ssh server and rdiff-backup
storage01 -- local node with data for duplicity
storage02 -- remote node with ssh server (for duplicity)
Full local backup::#duplicity
Local test :
docker run -ti --hostname data01 debian:latest
root@data01:/# apt-get update && apt-get -y upgrade && apt-get -y install duplicity
root@data01:/# cd $(mktemp -d)
Now, I'll copy one of my old project from production server using bash pseudo iface /dev/tcp (netcat/socat or ssh is better for real life):
root@data01:/tmp/tmp.YMFLVKqr67# tar xvzf - < /dev/tcp/test.roundside.com/5739
root@data01:/tmp/tmp.YMFLVKqr67# du -hs;find . -type f | wc -l;find . -type d | wc -l
512M .
33599
19637
So, this directory has 33599 files, 19637 directories and takes up 512MB of space.
root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ file:///var/duplicity/
Import of duplicity.backends.giobackend Failed: No module named gio
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1417922856.64 (Sun Dec 7 03:27:36 2014)
EndTime 1417922891.19 (Sun Dec 7 03:28:11 2014)
ElapsedTime 34.55 (34.55 seconds)
SourceFiles 53236
SourceFileSize 443922301 (423 MB)
NewFiles 53236
NewFileSize 443922301 (423 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 53236
RawDeltaSize 362387325 (346 MB)
TotalDestinationSizeChange 218640180 (209 MB)
Errors 0
-------------------------------------------------
real 0m36.289s
user 0m30.549s
sys 0m4.578s
First local backup, time: 36 seconds.
Same test with rdiff-backup (another container):
Full local backup::#rdiff-backup
root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ /var/rdiff-backup/`
real 0m32.287s
user 0m17.980s
sys 0m11.658s
Rdiff-backup, 32 seconds for initial local backup.
Remote backups
Lets make full backup to remote node (using ssh):
Full remote backup::#duplicity
Warning: duplicity 0.6.18 has a known 'bug (in result of some python library)' that produce
BackendException: ssh connection to host:22 failed: Unknown server host
This is because duplicity can't read ecdsa public key from known_hosts file.
You can disable ecdsa HostKey on server, export custom options for duplicity ssh connection (-o HostKeyAlgorithms) or (the better way) update duplicity. Debian backports repository has 0.6.24 version, that works perfectly.
root@data01:/tmp/tmp.YMFLVKqr67# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ ssh://storage01//var/duplicity/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1418780684.21 (Wed Dec 17 01:44:44 2014)
EndTime 1418780731.53 (Wed Dec 17 01:45:31 2014)
ElapsedTime 47.32 (47.32 seconds)
SourceFiles 53236
SourceFileSize 443979645 (423 MB)
NewFiles 53236
NewFileSize 443979645 (423 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 53236
RawDeltaSize 362387325 (346 MB)
TotalDestinationSizeChange 218640081 (209 MB)
Errors 0
-------------------------------------------------
real 0m50.069s
user 0m38.467s
sys 0m5.536s
Result: 50 seconds
Note: pay attention to duplicity syntax for remote node
ssh://storage01//var/duplicity/ - meant /var/duplicity on remote node
ssh://storage01/var/duplicity/ - meant $HOME/var/duplicity on remote node
Full remote backup::#rdiff-backup
root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ root@storage02::/var/rdiff-backup/
real 0m57.252s
user 0m7.235s
sys 0m2.590s
Result: 57 seconds
Note: for remote backups same version of rdiff-backup must been installed on the remote node.
Incremental backups
Now is the time for incrementsl backups:
For adding some data I'll clone node.js repo in existing local directory:
root@data01:/tmp/tmp.YMFLVKqr67# mkdir userdata;cd ./userdata;git clone https://github.com/joyent/node.git
At this time a have 43795 files, 20474 directories and 757MB of disk space.
#1 Incremental local backup::#duplicity
root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ file:///var/duplicity/
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20141216T213246Z.sigtar.gz to local cache.
Copying duplicity-full.20141216T213246Z.manifest to local cache.
Last full backup date: Tue Dec 16 21:32:46 2014
--------------[ Backup Statistics ]--------------
StartTime 1418785790.38 (Wed Dec 17 03:09:50 2014)
EndTime 1418785815.03 (Wed Dec 17 03:10:15 2014)
ElapsedTime 24.65 (24.65 seconds)
SourceFiles 64271
SourceFileSize 677202919 (646 MB)
NewFiles 11036
NewFileSize 233227370 (222 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 11036
RawDeltaSize 229606454 (219 MB)
TotalDestinationSizeChange 144816441 (138 MB)
Errors 0
-------------------------------------------------
real 0m26.750s
user 0m23.144s
sys 0m2.867s
Result: 27 seconds
#1 Incremental remote backup::#duplicity
root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ ssh://172.17.0.10//var/duplicity/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Dec 17 02:05:04 2014
--------------[ Backup Statistics ]--------------
StartTime 1418785633.50 (Wed Dec 17 03:07:13 2014)
EndTime 1418785667.86 (Wed Dec 17 03:07:47 2014)
ElapsedTime 34.36 (34.36 seconds)
SourceFiles 64271
SourceFileSize 677202919 (646 MB)
NewFiles 11036
NewFileSize 233227370 (222 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 11036
RawDeltaSize 229606454 (219 MB)
TotalDestinationSizeChange 144816441 (138 MB)
Errors 0
-------------------------------------------------
real 0m35.889s
user 0m30.717s
sys 0m3.750s
Result: 36 seconds
#1 Incremental local backup::#rdiff-backup
root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ /var/rdiff-backup/
real 3m54.791s
user 0m22.843s
sys 0m6.806s
Result: 3 minutes and 55 seconds
#1 Incremental remote backup::#rdiff-backup
root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ root@storage02::/var/rdiff-backup/
real 4m14.100s
user 0m16.245s
sys 0m2.813s
Result: 4 minutes and 14 seconds
Another backup, now some data was removed:
root@data01:/tmp/tmp.YMFLVKqr67# rm -rf ./addons
Local data has: 42653 files, 20030 directories and 731MB of space
#2 Incremental local backup::#duplicity
root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ file:///var/duplicity/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Dec 16 21:32:46 2014
--------------[ Backup Statistics ]--------------
StartTime 1418837027.28 (Wed Dec 17 17:23:47 2014)
EndTime 1418837041.33 (Wed Dec 17 17:24:01 2014)
ElapsedTime 14.06 (14.06 seconds)
SourceFiles 62686
SourceFileSize 652525867 (622 MB)
NewFiles 2
NewFileSize 8192 (8.00 KB)
DeletedFiles 1585
ChangedFiles 1
ChangedFileSize 649680 (634 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1588
RawDeltaSize 649808 (635 KB)
TotalDestinationSizeChange 241702 (236 KB)
Errors 0
-------------------------------------------------
real 0m14.372s
user 0m11.946s
sys 0m2.208s
Result: 14 seconds
#2 Incremental remote backup::#duplicity
root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ ssh://storage01//var/duplicity/
Synchronizing remote metadata to local cache...
Copying duplicity-inc.20141217T020504Z.to.20141217T030713Z.manifest to local cache.
Copying duplicity-new-signatures.20141217T020504Z.to.20141217T030713Z.sigtar.gz to local cache.
Last full backup date: Wed Dec 17 02:05:04 2014
--------------[ Backup Statistics ]--------------
StartTime 1418837112.75 (Wed Dec 17 17:25:12 2014)
EndTime 1418837127.06 (Wed Dec 17 17:25:27 2014)
ElapsedTime 14.31 (14.31 seconds)
SourceFiles 62686
SourceFileSize 652525867 (622 MB)
NewFiles 2
NewFileSize 8192 (8.00 KB)
DeletedFiles 1585
ChangedFiles 1
ChangedFileSize 649680 (634 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1588
RawDeltaSize 649808 (635 KB)
TotalDestinationSizeChange 241702 (236 KB)
Errors 0
-------------------------------------------------
real 0m15.151s
user 0m12.485s
sys 0m2.359s
Result: 15 seconds
#2 Incremental local backup::#rdiff-backup
root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ /var/rdiff-backup/
real 0m57.376s
user 0m15.506s
sys 0m2.582s
Result: 57 seconds
#2 Incremental remote backup::#rdiff-backup
root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ root@storage02::/var/rdiff-backup/
real 0m54.370s
user 0m3.627s
sys 0m2.042s
Result: 54 seconds
And the last incremental backup, now - adding two .iso files (23MB each).
As result: 42655 files, 20030 directories and 777MB of space.
#3 Incremental local backup::#duplicity
root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ file:///var/duplicity/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Dec 16 21:32:46 2014
--------------[ Backup Statistics ]--------------
StartTime 1418839612.96 (Wed Dec 17 18:06:52 2014)
EndTime 1418839628.26 (Wed Dec 17 18:07:08 2014)
ElapsedTime 15.31 (15.31 seconds)
SourceFiles 62687
SourceFileSize 700756267 (668 MB)
NewFiles 4
NewFileSize 48242688 (46.0 MB)
DeletedFiles 1
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 5
RawDeltaSize 48234496 (46.0 MB)
TotalDestinationSizeChange 29748287 (28.4 MB)
Errors 0
-------------------------------------------------
real 0m15.509s
user 0m13.164s
sys 0m2.239s
Result: 16 seconds
#3 Incremental remote backup::#duplicity
root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ ssh://storage01//var/duplicity/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Dec 17 02:05:04 2014
--------------[ Backup Statistics ]--------------
StartTime 1418839573.08 (Wed Dec 17 18:06:13 2014)
EndTime 1418839595.15 (Wed Dec 17 18:06:35 2014)
ElapsedTime 22.07 (22.07 seconds)
SourceFiles 62687
SourceFileSize 700756267 (668 MB)
NewFiles 4
NewFileSize 48242688 (46.0 MB)
DeletedFiles 1
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 5
RawDeltaSize 48234496 (46.0 MB)
TotalDestinationSizeChange 29748287 (28.4 MB)
Errors 0
-------------------------------------------------
real 0m22.801s
user 0m19.203s
sys 0m3.067s
Result: 23 seconds
#3 Incremental local backup::#rdiff-backup
root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ /var/rdiff-backup/
real 0m16.580s
user 0m13.669s
sys 0m1.922s
Result: 16 seconds
#3 Incremental remote backup::#rdiff-backup
root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ root@storage02::/var/rdiff-backup/
real 0m21.882s
user 0m5.306s
sys 0m1.998s
Result: 22 seconds
Restoring
Time to recover our data.
Oops: root@data02:/tmp/tmp.5m5JUUohRz# rm -rf ./* (I'll do this after each recover)
Full restore from local backup::#rdiff-backup
root@data02:/# time rdiff-backup --restore-as-of now /var/rdiff-backup/ /tmp/tmp.5m5JUUohRz/
real 0m57.613s
user 0m14.083s
sys 0m14.696s
Result: 58 seconds
Full restore from remote backup::#rdiff-backup
root@data02:/# time rdiff-backup --restore-as-of now root@storage02::/var/rdiff-backup/ /tmp/tmp.5m5JUUohRz/
real 1m34.002s
user 0m11.524s
sys 0m6.837s
Result: 1 minute and 34 seconds
Full restore from local backup::#duplicity
root@data01:/tmp/tmp.YMFLVKqr67# time duplicity --no-encryption file:///var/duplicity/ /tmp/tmp.YMFLVKqr67/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Dec 16 21:32:46 2014
real 0m35.601s
user 0m21.941s
sys 0m8.692s
Result: 36 seconds
Full restore from remote backup::#duplicity
root@data01:/# time duplicity --no-encryption ssh://storage01//var/duplicity/ /tmp/tmp.YMFLVKqr67/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Dec 17 02:05:04 2014
real 0m45.229s
user 0m35.745s
sys 0m9.431s
Result: 45 seconds
Restoring to specific time:
Restore from specific local backup::#rdiff-backup
root@data02:/# time rdiff-backup --restore-as-of 2014-12-17T17:34:04Z /var/rdiff-backup/ /tmp/tmp.5m5JUUohRz/
real 0m31.792s
user 0m15.487s
sys 0m11.008s
Result: 32 seconds
Restore from specific remote backup::#rdiff-backup
root@data02:/# time rdiff-backup --restore-as-of 2014-12-17T17:38:01Z storage02::/var/rdiff-backup/ /tmp/tmp.5m5JUUohRz/
real 1m30.043s
user 0m11.345s
sys 0m6.923s
Result: 1 minute and 30 seconds
Restore from specific local backup::#duplicity
root@data01:/# time duplicity --no-encryption -t 2014-12-17T17:23:47Z file:///var/duplicity/ /tmp/tmp.YMFLVKqr67/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Dec 16 21:32:46 2014
real 0m25.768s
user 0m17.950s
sys 0m6.917s
Result: 26 seconds
Restore from specific remote backup::#duplicity
root@data01:/# time duplicity --no-encryption -t 2014-12-17T17:25:12Z ssh://storage01//var/duplicity/ /tmp/tmp.YMFLVKqr67/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Dec 17 02:05:04 2014
real 0m46.186s
user 0m38.004s
sys 0m10.209s
Result: 46 seconds
Results and conclusion
I think this is enough. I've been made many tests, starting from 33599 files, 19637 directories and 512MB of space to 42653 files, 20030 directories and 731MB of space.
This chart shows how long Rdiff-backup and Duplicity backup process was (lower is better)
As you see, rdiff-backup is extremely slow in incremental back-up with many files and in restoring data. But a little bit faster in initial local backup.
But rdiff-backup has another disadvantage - occupied space
All duplicity backup data takes about 387MB, when the same backup in rdiff-backup - 880MB.
So rdiff-backup occupies 127% more space than duplicity.
The reason for this - duplicity store all data in compressed files called volumes. This can save a lot of space on disk.
But this gup is narrowing when you remove some data from local storage, in this case rdiff-backup will compress the difference and move this files into compressed snapshot.
While all data in duplicity backup are compressed already, and may be difficult access duplicity backup as usual files (which is easy in rdiff-backup), but I think is is a rare exception :)
Comparison of used space (Rdiff-backup and Duplicity, MB)
Initial backup
#1 Increment
#2 Increment
#3 Increment
That's all folks.
Links:
Duplicity
Rdiff-backup
Chartist-js