Last updated on Wednesday 30th of March 2022 09:18:49 AM

©XSIBackup-Pro OneDiff

 Please note that this post is relative to old deprecated software ©XSIBackup-Classic. Some facts herein contained may still be applicable to more recent versions though.

For new instalations please use new ©XSIBackup which is far more advanced than ©XSIBackup-Classic.

A feature exclusive to ©XSIBackup-Pro (deprecated). OneDiff stores changed blocks between backups to minimize backup transfered data. This allows to keep huge VMs backed up in seconds and consolidated in the target datastore. Unlike backups made by using Rsync, OneDiff does not need to calculate checksums, it knows which data has to be transfered and does it straight to the target backup location. Of course all OneDiff operations are performed hot, while the VM is running and users are connected.

 OneDiff algorithm exchanges snapshot information between hosts. It has been designed to work among uniform systems (same VMFS & Virtual HW version). There are different ways to make sure source and target VMs are compatible, the easiest one is to use identical ©ESXi versions (including build number).


OneDiff creates a mirror VM with the _XSIBAK string appended to the original VM name. Subsequent differential data is sent to this mirrored VM and integrated with base data sent there before. OneDiff is available to ©XSIBackup-Pro users.

From a conceptual point of view, this technology is very similar to CBT (Change Block Tracking) technology offered by ©VMWare through its propietary API (CBT is now supported natively in latest ©XSIBackup). The same goal is achieved by using higher level constructs like snapshots. Snapshot management is very solid in ESXi and thus you can expect a very reliable behaviour from part of OneDiff. In fact ©VMWare's CBT is nothing but the same, as changed blocks are stored in an additional file with a propietary format.



WHAT CAN IT ACHIEVE?

When it comes to backup VMs, size matters. The bigger the VM, the more time consuming and error prone the operation will be, not to mention resources taken by the backup operation (CPU, bandwidth, I/O, etc...) which won't be available to the running VMs. This is especially critical in 24x7 systems, or servers that can't just be stopped during the backup windows. If you want to backup your VMs off-site to be safe from thefts, fires, floods, etc..., then optimizing resource utilization becomes critical.

© copy;OneDiff has been conceived to make VM backup as light and fast as possible, by copying just changed bytes from your running VM to the _XSIBAK mirror. Once they get to the mirror system, they are integrated into a single consolidated disk file per .vmdk disk.

How do you know that the integration process succeeded and that your backups are consistent?

OneDiff performs two possible checks per -flat.vmdk file. The first check is performed by default, it's called Trivial Check. It's not called Trivial because it's useless, but just because not the whole file is checked. Only file size and the first N(configurable) megabytes are. When we check size and the first, let's say 500 mb, of a HD and compare them to the same extent on the backed up disk, taking for granted that there aren't any copy or transmission errors, we can be pretty sure, up to some point, that the files are identical. This is due to the fact that FS layout is generally coded at the beginning of a disk, so it can only be checksum identical to its counterpart if the FS layout is the same. Of course Trivial Check is better than nothing, but it's not 100% reliable. It's strongest point is that it can detect errors very fast, without the need to checksum the full disk size.

OneDiff backup e-mail report examples
OneDiff (Trivial Check) OneDiff (Certified)


If you want 100% reliability, then you can perform a full hash comparison on your -flat.vmdk disks by using the --certify-backup option. As with other backup programs inside ©XSIBackup, this will make sure sha1 or md5 hashes of your disks are the same. Still, when combined with ©OneDiff the consumed time is so short, that is faster than a regular backup and, of course, the transferred data is just the accumulated diff data since the last backup cycle.


HOW DOES IT WORK?

When you set the parameter --backup-prog=OneDiff in your argument list, ©XSIBackup-Pro will use the described instant diff backup feature, but the first time the backup is run, it will make a full backup of your virtual machine. This will prepare the base disks that will be used as the base of the following subsequent backups, to merge the diff data with. The folder will be named after the original VM, thus you must use a different datastore to store your clone copy.

Once the first full backup is made, ©XSIBackup-Pro will add the cloned VM to the inventory as YOURORIGINALVM_XSIBAK and this new automatically generated VM will be ready to be run by just starting it up. Nevertheless you should not start this VM unless you want it to replace the original or reset the OneDiff differential backups chain, as the .vmdk CIDs will change and the following time you run your OneDiff job this fact will be detected and ©XSIBackup-Pro will make an initial full backup restarting the process. If you feel curious and you want to test your piled backups are consistent, simply copy the full folder somewhere else, add the copy to the ESXi inventory and delete it after you have satisfied your curiosity. Take on account that the first time you initiate a OneDiff backup all snapshots will be removed, thus if you want a backup that can be compatible with test VMs or VMs that need to keep pre-existing snapshots, do not use OneDiff as your --backup-prog, leave this parameter unset and make a full backup of those VMs instead. OneDiff has been designed to backup big production VMs, not to be compatible wih test environments.

Since the first full backup and on ©XSIBackup-Pro will only transfer the changed blocks between the two VMs, minimizing data transfer. This way you can mirror a terabyte VM in just a few seconds, or minutes if you are using a narrow band connection like an ADSL. If this is your case, you can even make the first OneDiff backup locally and then move the VM physically to the other end of the narrower Internet connection. Please, take into account that I am using the term narrower to refer to "not so wideband" connections in opposition to fiber optics or similar Ultra Wide Band technologies. Obviously you should not try this through a modem connection. The time taken will greatly vary depending on various variables like: changed data, bandwidth, latency, reliability of the IP link, etc...

The primary copy of your VM being backed up will always run on top of a snapshot called xsibackupdiff. This snapshot will subsequently increase its Id number on every OneDiff cycle. On the other side, your secondary copy ending with _XSIBAK will always run on top of the base disks.

Should you need to make use of your backup copy, you can switch it on from its _XSIBAK copy. You can change the name and remove the _XSIBAK suffix, in fact this is only appended to the name, not the VM file dirs, which will remain named after the original. That is why you cannot keep a mirrored _XSIBAK VM in the same datastore as its original counterpart.

Needles to say that this backup method is great to keep mirrored copies of big VMs over narrow links ready to be used, but it's not so good in regards to recovery purposes, as every mirrored VM will be an exact unique copy of the first and any error, data corruption or virus will be rapidly transmitted. Thus you should always keep a log of backups that allows you to go back in time if needed. You can use a deduplicated device to handle many individual backups in the least space possible.

This post "One Diff Retention Policy" offers details about how to set up a OneDiff backup in combination with a historic archive that will allow you to restore a VM to a given point in time.

Error checks:

©XSIBackup uses different techniques to make sure that a mirror VM copy is correct. The most accurate one is parsing --certify-backup=yes|md5|sha1 to the backup job, this will calculate checksums for each of the -flat.vmdk files, certifying that files on both ends are identical. When not using --certify-backup, which is time consuming, especially for big disks, OneDiff performs a series of quick checks, basically checking the size of the file and comparing the first "n" megabytes (configurable in conf/xsiopts) of files on both ends. As the superblock is generally contained at the beginning of the disk, and it contains information about the file system, checking that the first "n" megabytes are identical, does not guarantee that the whole disk is identical on both sides, but on the contrary checking that it is not, certifies that there is a mismatch.