©XSIBackup  Using backup programs

©XSIBackup has become somewhat extense nowadays, and some clients have told us they feel a bit overwhelmed by so many options and combinations, especially in everything concerning to the --backup-prog argument.

Most of the --backup-prog options are very straight forward, nevertheless, there are some combinations of the --backup-prog with the --backup-point argument that will produce different results. Let's see which are all the options available and how they should be used.

Vmkfstools:

It is the default binary used to copy files between data-stores, it is fast and reliable and is present by default at every ESXi server. Nevertheless it can only copy data locally and it does not offer any de-duplication nor differential mechanism and can't copy snapshots either. Although XSIDiff can be even faster than vmkfstools It will remain by now the base binary for full local backups.

As vmkfstools can't copy snapshots. If you want to preserve any pre existing one in a local backup datastore => datastore, you must use Rsync or XSIDiff as the --backup-prog value.

XSITools:

This backup program will copy data to an XSITools repository, where this data will be stored de-duplicated, thus the XSITools repository is able to multiply storage capacity by many times. It is also able to detect holes or empty blocks of data to avoid copying just zeros, so it reaches a high throughput ratio over the whole files being transferred. It is a very convenient way to store data. ©XSITools can store files to a local datastore or to a remote ESXi or Linux server.

You can restore files in any XSITools repository by using the --restore module. Read more...

Borg (go to Borg official site):

Borg backup program allows your to backup VMs to a Borg repository. Borg servers are just a Linux/ Unix computer with the Borg Backup binary installed. Unfortunately it's not possible to run a true Borg client in ESXi nowadays, thus ESXi sends ALL DATA to the Borg server and the de-duplication process is performed at the server side, therefore, this backup program is only suitable to be used in LAN environments, and not to backup over a WAN or narrower Internet connection. Borg backup program can only be combined with an IP:port:/path/to/borg/repo in the --backup-point argument all Borg repositories are reachable only over an IP:port. You can restore VMs from any Borg repository by using the --restore module.

Rsync (go to Rsync official site):

This is the well known Rsync differential copy program used by so many people around the globe and based on the Delta algorithm. It is great at reducing traffic figures, but it's slow when copying files over SSH. It's not Rsync's fault, this is due to reduced buffer size, limited to 4096 bytes when copying data over stdout => stdin and also to overhead caused by slow encryption algorithms. It's also probably "too efficient" at calculating differential data, probably copying redundant bytes would be beneficial when working with huge VM files, this is due to their particular characteristics: big size fixed length, mostly aligned from one backup to the following. To compensate some of these weaknesses we have introduced XSIDiff, which will take care, from version 10.0.0 and on, to perform first transfers of data over IP, this will overcome Rsync slowness, then on second runs and above, both in direct Rsync synchronizations and OneDiff backups, Rsync will perform the actual differential data synchronization, as it's unbeatable in doing this.

OneDiff:

OneDiff is ©XSIBackup way to make a differential backup locally or over IP, thus it is compatible with local and remote paths. ©XSIBackup-Pro will store blocks written between backup cycles in a snapshot file. This way, when the subsequent backup time arrives, ©XSIBackup-Pro will take these blocks, move them directly to the mirror _XSIBAK Virtual Machine and integrate them with the pre-existing data. As OneDiff uses snapshots to store incremental data, it will be as reliable as your snapshot system is, this means that:

1 - You will need your VMs to be in a healthy state. Please do note that a running VM is not proof that it is in a healthy and backup ready condition, the snapshot mechanism must be in a reliable working state.

2 - ©VMWare Tools will need to be installed in the guest OS, along with any other related service needed to eventually quiesce applications running in the guest. Quiescing can be a somewhat obscure subject for beginners. You only need to quiesce a VM if it's running a database server or some other software that needs to ensure data is flushed to disk consistently prior to taking a snapshot. You can easily overcome this by just using --backup-how=warm option. Thus quiescing is something that you will have to pay attention to in cases in which you are running database servers that are serving users' requests while the quiescing process is taking place and you cannot afford to stop the guest OS for not even some seconds. Windows servers can be more difficult to quiesce, but you can always work the problem around in various different ways. Start to read here for more details.

3 - MBR Windows based systems are not suitable to host applications that require quiescing to ensure their data is consistent after a hot backup. Then, if you have some Windows MBR system with, let's say, Exchange installed, you will need to previously convert its boot record to GPT.

OneDiff is compatible with regular backups. That is, you can engage a given VM in a OneDiff backup. That VM will be running on top of a snapshot called xsibackupdiff, if you then perform a regular backup on that VM, another snapshot will be added on top of xsibackupdiff and the virtual machine will be copied in one of the following ways:

1 - If you perform a local backup with vmkfstools, the VM will be cloned from it, thus it will include all data, including data inside the xsibackupdiff snapshot file. This means your snapshot will be integrated with the base disks at the backup target location. But once the backup finishes, the backup snapshot file will be deleted and the VM will still run on top of the xsibackupdiff snapshot, so you will be able to retake the OneDiff cycle as if nothing would have happened.

2 - If you perform a local backup by means of Rsync or XSIDiff, ©XSIBackup will generate a new snapshot on top of xsibackupdiff just like above, but it will copy all files as are, including the xsibackupdiff snapshot files. Thus, the resulting VM will preserve the snapshot structure below the backup snapshot, in this case the OneDiff snapshot xsibackupdiff. Once finished, the backup snapshot will be deleted and the remaining structure will still be compatible with the OneDiff cycle initiated on that VM. The next OneDiff round the xsibackupdiff snapshot files will be copied to the mirror VM and integrated with the pre-existing data sent there on the previous OneDiff cycles.

3 - If you do an IP backup by means of XSIDiff or Rsync, the same as above will happen, with the only exception of files being copied over an IP link instead of two local paths. The result will be that you can continue to use the already started OneDiff cycle.

XSIDiff:

This is the newest addition to the repertoire of backup programs. XSIDiff is a standalone binary that requires its own license key and that may be used separately to copy files manually between datastores, different ©ESXi hosts or from an ©ESXi host to a Linux system. The standard individual license allows you to copy data to any host and to receive data from any host equipped with the un registered version of XSIDiff.

Since version 10.0.0 it will overtake the initial transfers made by means of Rsync so far. Rsync is far slower than XSIDiff at the time of doing that first full copy of the vmdk disks. XSIDiff is able to just copy VMFS used blocks, thus saving a lot of time. On top of that it's also faster copying raw data locally and over a network.


Daniel J. García Fidalgo