©XSIBackup: Rsync Considerations

Please note that this post is relative to old deprecated software ©XSIBackup-Classic. Some facts herein contained may still be applicable to more recent versions though.

For new instalations please use new ©XSIBackup which is far more advanced than ©XSIBackup-Classic.

IMPORTANT: Rsync is a well known data sync program which was offered as part of ©XSIBackup-Free Classic (deprecated) to be an alternative to copy your VMs over IP.

If you are an ©XSIBackup-Pro/DC user, any of the available programs in the Classic Pro version (superceded by DC Edition as of 2021-01-07) is by far faster than Rsync. DC Edition is in turn faster than Classic Pro binaries. You may nevertheless still find Rsync useful when it comes to transfer medium sized -flat.vmdk files manually on idle VMs.

Download ©XSIBackup
Download Rsync 3.1.0

Rsync was a third party dependency to ©XSIBackup since version 4.1.0. ©XSIBackup-rsync is a plain rsync 3.1.0 compiled for ©VMWare ©ESXi, you can download a plain Rsync 3.1.0 for ©ESXi here download Rsync for ©ESXi. The only reason why the binary was renamed to xsibackup-rsync was to make lists more readable and to ease changing file permissions. You can download it's source code from its official URL here: https://rsync.samba.org/ftp/rsync/src/ and you can find information on how to compile software for the ©ESXi OS at ©VMWare's website and at this article at www.virtuallyghetto.com.

By using Rsync along with ©XSIBackup you can take advantage of its Delta Algorithm implemented by Rsync. However some considerations must be taken into account before we start to jump for joy.

Rsync is a great piece of software, no doubt about that, it's a general purpose file sync utility though. The Delta Algorithm implemented by Rsync calculates the differences for the files on both ends without making any assumption on the size of the files or its constituent blocks, thus it is slow when compared to software specifically designed for ©ESXi backups, like ©XSIBackup.

XSIBackup-rsync functioning

The price to pay:

Comparing the two files is CPU intensive and takes a while depending on the available CPU you have at your ©ESXi host and on the size of the file, thus you might find yourself in such a situation that comparing the two .vmdk files on both ends of your mirrored ©ESXi system and transferring the changed blocks might take longer than actually transferring the whole .vmdk file.

A Delta Algorithm is a way to use the CPU of your hosts to compute the file differences and save bandwidth and transfer time. This is where ©XSIBackup comes in handy. It has been designed to backup VMs at ©ESXi hosts.

It not only takes care to snasphot and quiesce the virtual machines to allow hot backups, it is also zero aware, namely: it knows which blocks can be skipped as they contain nothing. It also knows which were the constituent blocks of the virtual disks the last time they were backed up. Should that information be lost, they will be recalculated, that isn't neccessary most of the times though. The more it knows on advance, the less CPU it will use and the faster the whole process will be.

You are probably thinking: "ohhh!, but isn't there any other way to just know which blocks have to be sent whithout performing any pre-processing?".

Yes, there is: Change Block Tracking technology offered by ©VMWare, which ©XSIBackup-DC offers even for unlicensed versions of the ©ESXi Hypervisor.

How do Rsync transfers work.

The Rsync client in your ©ESXi host needs a counterpart binary in the other end of the copy channel to act as an Rsync server. They can communicate with each other through basically two main methods: SSH tunnel or Rsync protocol over TCP/IP. The latter will be faster as you won't need to encrypt the data through an additional auxiliary process, it will pose security concerns though, as the data won't be encrypted.

You can assume that encryption will reduce the transfer efficiency by some 25%, depending on the type of hardware you have.

Security concerns, compiling xsibackup-rsync on your own.

Some people have posed their concerns about installing an alien binary in their ©ESXi hosts, in regards to the companion binary xsibackup-rsync. We do understand their doubts, as nowadays the net has become a really doubtfull (please allow me to coin this neologism by doubling the "l", I find it to be more precise) place in almost any way that you can imagine. We obviously must give a proper answer to those doubts, and I will break them down into this two points:

1 - The binary xsibackup-rsync is not needed unless you want to replicate VMs over TCP/IP when using some old version of our software. If you only want to backup your VMs to a local datastore you can get rid of it, you don't even need to install it. ©XSIBackup itself is a script so what it does is available there to be inspected by anybody that wants to. We managed the posibility to include adds in the e-mail report and reserved a place in the layout, but we still haven't done anything in regards to this (UPDATE: from v 4.6.0 we have placed a banner encouraging to make a donation to get extra features). In any case you could always remove them if they bother you, as the license allows you to modify the source code.

2 - If you still want to use xsibackup-rsync but don't feel comfortable with the provided binary or your enterprise security policy forces you to, you can compile your own. You will only need rsync 3.1.0 available at https://download.samba.org/pub/rsync/src/rsync-3.1.0.tar.gz and CentOS 5.3 32 bit available at: http://ftp.ksc.co.th/CentOS/5.3/. CentOS 64 bit versions might work as well, ©ESXi has some limitations in regards to some 64 bit system calls that might make your compiled Rsync binary unusable. Of course you will need to install all staff related to development, compilers, c headers, etc...

A common developer installation should do it. We have found CentOS 5.3 32 bit to be the best option for ©ESXi 5.5 and 6.0, though you can find some nice articles in the net, one of the nicest by William Lam is here:

http://www.virtuallyghetto.com/2011/02/how-to-compile-statically-linked-rsync.html. In any case you have all the information you need to compile Rsync for ©ESXi at VMWare's site http://www.vmware.com and there are a number of articles available on how to compile drivers, which is more or less the same http://www.v-front.de/2013/03/how-to-build-device-drivers-for-esxi-5x.html.

If anybody still feels uncomfortable about installing xsibackup-rsync all that we can say is: whatever you do in life, wherever you go, whoever you meet, sooner or later you will have to trust somebody, it's up to you to decide who.

Daniel J. García Fidalgo
33HOPS