Last updated on Monday 2nd of May 2022 07:56:10 PM

©XSIBackup CBT Implementation

Instant differential backups with multitenancy distribution of copies

(*) Do not forget to read the CBT documentation in the ©XSIBackup-DC manual.

©XSIBackup-DC introduces CBT support in version 1.5.0.0 as part of ©XSIBackup-DC edition. You can try the feature in the Free version during the trial time.

©VMWare's CBT technology allows to know the blocks that have changed since some moment to directly transmit those blocks on differential backups, thus achieving a much faster ©ESXi VM backup and avoiding the overhead caused by some delta algorithm, which would require hashing each block to detect the changed ones.

The data that allows us to know which is the state of some particular CBT differential transfer is a sequential integer number which grows as the data changes on disk. This numbers aren't correlative, they grow as the VM eveolves in time though. Thus some bigger integer denotes some more recent state.

This is a great advance, as it allows to drastically reduce the time employed in differential replicas and backups, which is specially interesting in case of huge VMs, which would take hours or days just to have its deltas calculated.
Multiple end point CBT replication scheme

How does our CBT implementation work

We keep a set of hidden configuration files for each VM in your ©ESXi host which is engaged in a CBT scheme. Those files are stored in the .xsi/.cbt folder under your VM directories hosting virtual disks. they are the following:
  • One file per target/disk holding the sequence numbers to each target replica:

    e.g.: .xsi/.cbt/.seq-WS2012-flat.vmdk

    This file contains lines like the following:

    /vmfs/volumes/backup2/replicas/WS2012; 415
    192.168.33.183:22:/vmfs/volumes/datastore1/WS2012; 1241
    256.41.34.121:8022:/vmfs/volumes/datastore4/replicas/WS2012; 8457

    The above example is for a seq- descriptor file for a flat file (WS2012-flat.vmdk) in a VM named WS2012 which is keeping three replicas in different locations: one local datastore, one storage server in a LAN and another storage server somewhere in the internet (fake address).

  • One cbt-history file per disk:

    e.g.: .xsi/.cbt/.seq-history-WS2012-flat.vmdk

    This file contains lines like the following:

    2021-04-25T14:28:31 -> /vmfs/volumes/datastore2/WS2012; 1
    2021-04-25T14:36:11 -> /vmfs/volumes/datastore2/WS2012; 5
    2021-04-25T14:37:27 -> 192.168.33.183:22:/vmfs/volumes/datastore1/WS2012; 1
    2021-04-25T14:55:11 -> 192.168.33.183:22:/vmfs/volumes/datastore1/WS2012; 5
    2021-04-25T14:58:30 -> /vmfs/volumes/datastore2/WS2012; 17
    2021-04-25T15:06:00 -> /vmfs/volumes/datastore2/WS2012; 21
    2021-04-25T15:07:46 -> 192.168.33.183:22:/vmfs/volumes/datastore1/WS2012; 7
    2021-04-25T15:10:20 -> /vmfs/volumes/datastore2/WS2012; 29
    2021-04-25T15:17:18 -> 192.168.33.183:22:/vmfs/volumes/datastore1/WS2012; 9
    2021-04-25T15:20:14 -> 192.168.33.183:22:/vmfs/volumes/datastore1/WS2012; 13
    2021-04-25T15:22:11 -> /vmfs/volumes/datastore2/WS2012; 45

    Each line represents one CBT operation on a file and target, so you can know when some differential operation was performed.
As you can see ©XSIBackup keeps track of the state for multiple sets of mirrored disks in multiple locations, no matter whether they are local or remote in some IP location.

Resetting CBT for a Virtual Machine

When you reset CBT for a given VM you do so completely. That means removing the information for all eventual replication points that you may have previously set. which would in turn cause ALL replication points to be fully synchronized on the next differential backup job.

When you want to reset some replication state for just one of the replicated VM nodes, you just have to edit the corresponding .seq- file for all the disks you want to resync. You will normally do so for all disks.

By setting the CBT sequence integer to 0, you will force the whole syncronization of the disk again to the target location.

192.168.33.183:22:/vmfs/volumes/datastore1/WS2012; 0

Please, note that on subsequent CBT differential backups the sequence numbers will not be reset to low integers, but will indeed continue using the numbers that were in use before resetting the sequence to 0.

Checking the replicas

©XSIBackup automatically strips all CTK file related information from the .vmx and .vmdk file descriptors of the resulting replicated VM. See the C option in --options to prevent this from happening automatically. Be aware that you use the C option you will most probably receive some errors when trying to switch the remote VM on. This is an advanced option that should not be used unless you know exactly why you need to. It's also possible to just allow ©XSIBackup to strip the CTK info to just add it to the replica when you need it.

Keep in mind that CTK operations for a VM are bound to one specific host and state of a VM. You can't just try to set a chain of CBT replicas up across multiple hosts, it's just not that trivial. You can nevertheless backup a replica, which is a common strategy in order to keep the load of a backup away from the production server.

Once you have performed a differential replica on some remote end for a virtual machine you cannot just plainly switch that VM on to verify that it is indeed working and that all expected data has been sent. If you do so, you will cause at least the system disk to be modified, ©XSIBackup will detect that on some subsequent run of the differential backup job and will resync the VM fully.

To prevent that from happening ©XSIBackup offers an option to automatically register the VM and take a test snapshot. This allows to just switch the VM and browse anything you need in the guest OS without affecting the base disks which are engaged in a CBT replication scheme.

You could of course do that manually for the replicated VM, ©XSIBackup can do that for your convenience by employing the "R" option though.

--options=R

Thus a CBT replica job with automatic registration of the remote VM will look like this:

./xsibackup --replica=cbt "VMs(WS2012)" root@192.168.3.83:22:/vmfs/volumes/datastore1 --options=R

On additional runs of the same job ©XSIBackup will take care to detect the test VM and snapshot and remove it prior to start the CBT differential backup. This way you can permanently keep a runnable replica that will allow to check the state of the differential backup.

Please do note that automatic registration and the test snapshot can only be created automatically when the backup is local (same ©ESXi host context) or when the IP target is another ©ESXi server. If you run a CBT replica to a Linux host over IP and you then want to try the replica out by registering it to some other ©ESXi host to which the Linux server is attached as NFS, you will need to create the test snapshot manually and discard it also manually once you finish to test the VM. You will then have to unregister the VM manually too.

Should you fail to do the above, the data will become out of sync, that condition will be detected by the backup job and a full VM transfer will be carried out on the following job cycle.

You must not delete the test snapshot on the replica under any circumstance, unless you are definitely dismantling the CBT replica. When you delete the test snapshot on a CBT replica through the ©ESXi GUI, you coalesce that data to the base disks and you break the synchrony between the source and target.

©XSIBackup's CBT replica algorithm does not "delete" the snapshot on the remote target, it discards or removes it. Again the ©VMWare ©ESXi terminology in regards to snapshots is counter intuitive, thus make sure that you don't misunderstand something. Use the forum in case you are in doubt.

Anatomy of a CBT replica

Replicas created by the --replica=cbt argument will end in _XSIREP. Only the name of the VM in the .vmx file will be changed. The rest of the VM anatomy will remain the same as in the original. Also remember that ©XSIBackup creates remote replicated VMs and backup folders by the original VM's name. Thus if you backup a replica, something that we highly recommend to keep the load away from a production server, it will be backed up as YOURVM_XSIREP. This doesn't place any additional constraint apart from the naming scheme itself.

The test snapshot is useless, it only serves the purpose of switching on the VM for visual inspection. The VM is plainly powered off with no downtime on the following --replica cycle, should it be on. There is not danger to damage the mirror VM, as it is running on top of this test snapshot, therefore the base disks, which are the real subject of the replicated data, remain untouched.

Resetting from a previous sequence point in time

As previously explained, CBT keeps track of changed blocks, even erased data. Therefore, just as long as you keep some remote replica in a consistent condition, you can easily resend all changed blocks from some previous state of the VM.

You can do so by inspecting the history files and setting the sequence number to some previous sequence integer. It doesn't matter if you pick some integer in between real values stored in the history files. The CBT differential backup will decay to the the closest sequence state below the provided number.

As explained above you can set the number manually in any of the .cbt- files for some particular disk/s, nonetheless, this is not very wise, as these files are read during the program execution. If for some reason you introduce some invisible character or mess them up some other way, you may be forced to reset the whole CBT scheme for all the end points.

To prevent this from happening, you can optionally pass the sequence number in the cbt directive of the --replica action argument this way:

--replica=cbt:27

Abobe the :27 indicates, retake CBT from sequence number 27 for the target especified in the job, or the full job...

./xsibackup --replica=cbt:27 "VMs(WS2012)" root@192.168.3.83:22:/vmfs/volumes/datastore1 --options=R

As you can see, our CBT implementation is profound and flexible, thus, you can cover almost any case scanario that you may face up with. It also aoffers the tools to recover from most issues you may encounter in your daily work with ©VMWare ©ESXi virtual machines.

Troubleshooting

VM autoregister issues

CBT can register your remote VM by issuing the --options=R argument. Registering a VM consists in adding the .vmx file to the inventory. In case you try to register some VM that is already registered, or which register information is cached, you may receive the following message:

A specified key, name, or identifier 'n' already exists
Do make sure that you have removed the VM completely before attempting to register it again. The CBT feature will take care to unregister the remote VM (_XSIREP) in case ir exists, nonetheless, you could fall into a situation in which the VM can't be unregistered or in which some information on its register is still available to the host service. In that case we recommend that you restart the managing agents, manually remove the VM from the inventory and then delete or rename all the VM folders.

/etc/init.d/hostd restart /etc/init.d/vpxa restart