Registered users
Linkedin Twitter Facebook Google+
  This website uses its own cookies or from third parties Close
33HOPS, IT Consultants
33HOPS ::: Proveedores de Soluciones Informáticas :: Madrid :+34 91 663 6085Avda. Castilla la Mancha, 95 - local posterior - 28700 S.S. de los Reyes - MADRID33HOPS, Sistemas de Informacion y Redes, S.L.Info

<< Return to index

XSIBackup-rsync Considerations

IMPORTANT: rsync is a legacy backup program offered as part of XSIBACKUP-FREE to be a free alternative to backup your VMs over IP. If you are an XSIBACKUP-PRO user, please, do not use Rsync, any of the available programs in the PRO version (borg, OneDiff, XSITools) is by far faster than Rsync.

Download XSIBackup at http://33hops.com/xsibackup-vmware-esxi-backup.html

Quickstart guide: http://33hops.com/blog_xsibackup-quickstart.html

Rsync is a third party dependency to © XSIBackup since version 4.1.0. XSIBackup-rsync is a plain rsync 3.1.0 compiled for VMWare ESXi (UPDATE: since version 4.6.3 Rsync has been modified to improve transfer speed), you can download a plain Rsync 3.1.0 for ESXi here download Rsync for ESXi. The only reason why it has been renamed to xsibackup-rsync is to make lists more readable and to ease changing file permissions. You can download it's source code from its official URL here: https://rsync.samba.org/ftp/rsync/src/ and you can find information on how to compile software for the ESXi OS at VMWare's website. By using Rsync along with XSIBackup you can take advantage of the "Delta Algorithm" implemented by rsync. However some considerations must be taken on account before we start to jump for joy.

Rsync is a great piece of software (that is for sure) but it does no magic, what I mean by that is that to apply the Delta Algorithm rsync must compare files on both ends bit by bit (well, not really every bit, a complex algorithm is used but all bits are implied), once it knows which chunks of data it has to send over the wire it starts to transfer them (only the changed blocks) minimizing data transfer... but as anything in life there's always a tradeof and a price you have to pay :-(

XSIBackup-rsync functioningThe price to pay:

Comparing the two files is CPU intensive and takes a while depending on the available CPU you have at your ESXi host and on the size of the file, thus you might find yourself in such a situation that comparing the two .vmdk files on both ends of your mirrored ESXi system and transferring the changed blocks might take longer than actually transferring the whole .vmdk file. This will more likely happen in a scenario in which you have not too mighty CPUs on your hosts and a lot of available bandwidth, like per instance a gigabit LAN with good network hardware. On the contrary if you have a mighty CPU and are planning to sync your ESXi boxes accross the Internet you will probably prefer to use the Delta Algorithm as it will be a lot faster. Take on account that the ESXi shell will only use one core, therefore, it is the processing capacity of a single core that will make the difference, having 32 cores available won't help.

You are probably thinking: "ohhh!, but isn't there a way to just know which blocks have to be sent?".
Yes, there is: OneDiff, which is offered as part of XSIBACKUP-PRO.

One important aspect to take on account is whether your VMDK files contain a lot of zeroes. This is valid for thin and thick provisioned disks. If so, using the --backup-prog with an additional parameter - z - will help you improve speed over narrower links:

Example 1: --backup-prog=rsync:z

What the above does is activate the minimum level of compression on the SSH tunnel. You don't need more than the minimum to detect and compress zeros. Elevating the compression level, on the other hand, will not improve transfer speed, but will soar CPU usage.

You should also consider attaching the target storage to your master ESXi box via NFS/iSCSI and doing a regular backup via the local file system, as this will be faster most of the times than transfering files over Rsync. This is not always possible and every environment is different so its up to your knowledge of your system to find the best way for you. In any case take on account that network storage like NFS shared by different ESXi hosts is the ideal environment to build resilient VMWare infrastructures. Thus, consider using asynchronuos NFS; but every environment is different and there's no rule of thumb, trial and error will be your best approach method, there are too many variables.

We have tweaked XSIBackup to use rsync the best way possible, we give the possibility through the program's command line arguments to choose whether to use Delta Algorithm or not.

The --backup-point argument in detail:

Example 2: --backup-point="192.168.0.100:22:/vmfs/volumes/datastore2"

In this case we will be transferring some VMs to a remote server to be mirrored in the remote directory /vmfs/volumes/datastore2. If this is the initial synchronization and thus the remote file does not exist, a full copy will be performed. On subsequent copies the delta algorithm will be used.

How do Rsync transfers work.

The Rsync client in your ESXi host (or xsibackup-rsync) needs a counterpart binary in the other end of the copy channel to act as an Rsync server. XSIBackup-Rsync, will look for this counterpart binary in the path specified by the xsidefaultpath variable, which is set at the beginning of the script.



If you don't have a /vmfs/volumes/datastore1 default datastore at your remote server, you may need to tweak this value to accomodate your needs.

Security concerns, compiling xsibackup-rsync on your own.

Some people have posed their concerns about installing an alien binary in their ESXi hosts, in regards to the companion binary xsibackup-rsync. We do understand their doubts, as nowadays the net has become a really doubtfull (please allow me to coin this neologism by doubling the "l", I find it to be more precise) place in almost any way that you can imagine. We obviously must give a proper answer to those doubts, and I will break them down into this two points:

1 - The binary xsibackup-rsync is not needed unless you want to replicate VMs over TCP/IP. If you only want to backup your VMs to a local datastore you can get rid of it, you don't even need to install it. XSIBackup itself is a script so what it does is available there to be inspected by anybody that wants to. We managed the posibility to include adds in the e-mail report and reserved a place in the layout, but we still haven't done anything in regards to this (UPDATE: from v 4.6.0 we have placed a banner encouraging to make a donation to get extra features). In any case you could always remove them if they bother you, as the license allows you to modify the source code.

2 - If you still want to use xsibackup-rsync but don't feel comfortable with the provided binary or your enterprise security policy oblies you to, you can compile your own. You will only need rsync 3.1.0 available at https://download.samba.org/pub/rsync/src/rsync-3.1.0.tar.gz and CentOS 5.3 32 bit available at: http://isoredirect.centos.org/centos/5/isos/i386/. Of course you will need to install all staff related to development, compilers, c headers, etc... A common developer installation should do the trick. We have found CentOS 5.3 32 bit to be the best option for ESXi 5.5 and 6.0, though you can find some nice articles in the net, one of the nicest by William Lam is here: http://www.virtuallyghetto.com/2011/02/how-to-compile-statically-linked-rsync.html. In any case you have all the information you need to compile Rsync for ESXi at VMWare's site http://www.vmware.com and there are a number of articles available on how to compile drivers, which is more or less the same http://www.v-front.de/2013/03/how-to-build-device-drivers-for-esxi-5x.html.

If anybody still feels uncomfortable about installing xsibackup-rsync all that we can say is: whatever you do in life, wherever you go, whoever you meet, sooner or later you will have to trust somebody, it's up to you to decide who.

Daniel J. García Fidalgo
33HOPS



Website Map
Consultancy
IT Manager
In Site
Resources & help
Index of Docs
33HOPS Forum

Fill in to download
The download link will be sent to your e-mail.
Name
Lastname
E-mail


            Read our Privacy Policy