How to use OneDiff and still keep a historic set of backups.

Please note that this post is relative to old deprecated software ©XSIBackup-Classic. Some facts herein contained may still be applicable to more recent versions though.

For new instalations please use new ©XSIBackup which is far more advanced than ©XSIBackup-Classic.

OneDiff is a good way to backup your VMs over a narrower network, like a WAN. Narrower does not mean a 33 kbps modem link, but a data link wide enough to move the amount of data you need in a reasonable time. Those are relative terms, and it's youwho must decide and calculate what you need.

As OneDiff only copies the changed bytes since last backup, most of the times you'll be able to keep your VMs mirrored to an offsite location in some minutes. Nevertheless, keeping a mirrored copy is not enough, why?, because in case something goes wrong with your data in the production VM, you'll be mirroring the problem to the other end. So, what to do?, you need a way to keep a set of backups on the other end, so that you can move in time to a date when your data was intact. The wider your retention policy is, the better. Take on account that sometimes problems do not show up instantly, so you may need to move back in time one week to recover your OS properly. I'm putting it in the worst of the cases to make it clear.

©XSIBackup-Pro Classic is flexible enough to offer you resources to cover this need. OneDiff allows you to sync changed blocks, and XSITools allows to store deduplicated compressed VMs. Let's see how to accomplish this.

First thing to do is to prepare a OneDiff backup job and add it to the xsibackup-cron file, you have probably already done this.

/vmfs/volumes/datastore1/xsi-dir/xsibackup backupid=01 --time="Mon 02:00|Wed 01:00|Sat 02:00" \
--backup-prog=onediff --backup-point="192.168.33.33:22:/vmfs/volumes/datastore1" \
--backup-type=custom --backup-vms="YOURVM" --mail-to=you@yourdomain.com --use-smtp=3 \
--on-success="backupid->02" --on-error="backupid->02"

O.K., this will make a OneDiff backup and inform you about its state once it has been completed. You should be familiarized with this type of backup job at this point, so we won't delve into it now.

Now you need a way to chain an XSITools backup on the other end once this first stage of the process is over. ©XSIBackup-Pro Classic allows to fire remote backup commands so that they are executed on another server via the --host flag. Thus we can add the following line to our backup logic:

/vmfs/volumes/datastore1/xsi-dir/xsibackup backupid=02 --host=192.168.33.33 --backup-prog=xsitools:z \
--backup-point="/vmfs/volumes/NAS03/xsit-$( date +'%Y-%m')" --backup-type=custom \
--backup-vms="YOURVM_XSIBAK" --mail-to=you@yourdomain.com --use-smtp=2 --override=xsibakfilter

Let's see the details:

--host=192.168.33.33, will make the current command be executed in the remote host with IP 192.168.33.33 where the OneDiff backup was stored by means of the previous command. You need to have previously linked the current server to the remote ESXi box 192.168.33.33, via the --link-srv command.

--backup-prog=xsitools:z, here you set XSITools as the program to be used to do an incremental backup. Note that :z has been appended to the xsitools keyword. This is recommended, you should always use it, as it reduces the backup by an average of 50% and does not affect backup speed.

--backup-point="/vmfs/volumes/NAS03/xsit-$( date +'%Y-%m')", the backup point argument contains the path to the NAS03 device, mounted as a datastore with a dinamic folder name as the last storage dir that will create folders such as xsit-2017-05. If you use dynamic folders, new XSITools repositories will be created as the calendar rolls.

--override=xsibakfilter, this argument will allow you to backup a OneDiff backup directly, which is, by default, filtered out from the VM listing and thus ignored.

If everything has been set correctly, the first backup job will perform a OneDiff backup to the server with IP 192.168.33.33, where a mirrored copy will be created after each backup cycle, ending in_XSIBAK. Once this first backup job has finished, the --on-error and --on-success arguments, set to execute the backup with backupid 02 will fire it, no matter what the exit status of the first job is. It's a must to try all backup jobs in the command line before adding them to the xsibackup-cron file.

The second backup job will perform a XSITools backup to the --backup-point="/vmfs/volumes/NAS03/xsit-$( date +'%Y-%m')" folder, where the VM will be appended to the previously stored backups. XSITools is capable to store a huge amount of data, thus you should be able to store many backups in your repository. In any case, you should read all the XSITools documentation and set some limits to prevent your repository from growing unlimitedly, in this example we used a dynamic folder to limit repositories in time.

We have chosen XSITools as the tool to archive our OneDiff backup, but there are various option available:

• Borg Backup: a good option if you have access to your Borg server within a gigabit LAN.

• XSINAS: same as Borg Backup. It has the advantage of offering a real time inline deduplicated FS, the drawback is that it is more resource hungry than Borg, it's less efficient and the backup speed will degrade as the FS fills.

• XSITools: it's probably your best bet. It's lightweight, it offers de-duplication, you can store your backups in a VMFS volume or in any other FS available through your NAS and can store a lot of data. Its weakest point is that the de-duplication technology used is less efficient than that of Borg Backup due mainly to its much bigger block size, still it multiplies your available backup room.

Daniel J. García Fidalgo
33HOPS