Last updated on Tuesday 16th of August 2022 03:17:34 PM

©XSIBackup Rotating Replicas

Different ways to rotate replicas

(*) Do not forget to start at Rotation documentation in the ©XSIBackup-DC manual.


Even though we have created a deduplication engine that allows to compress VM data up to 99% with a proprietary FS XSIFS that allows to recover any file in any of the unlimited restore points of a repo, we have detected that some users tend to accumulate sets of replicas instead of using the advantages of deduplication plus compression.

It all makes sense, as the replicas are ready to use and are somewhat simpler to understand from a conceptual point of view, thus they are the favourite of some power users or sys admins who just want to have a quick recovery method that avoids restoring from a repository.

We conceived ©XSIBackup-DC pointing high, still, in the end many of our users are not big enterprises willing to archive thousands of different restore points of their servers in dedicated backup servers, but smaller consultants or SMEs that just want to keep a limited set of ready to use copies.


You should learn to use backup sets, still we acknowledge the usefulness of replica sets.

When it comes to replicate data, the size of the VM is the key. Should the VM be 'big' (we'll leave the relativity of the term to your understanding and scenario) replicating the whole data every time may not be desirable or even possible.

With the introduction of our CBT implementation, keeping a set of replicas has now become a more sense making approach, as once you have some replica in place, transferring the differential blocks is a matter of seconds.

In this post we will cover two ways to keep sets of replicas, although one of them is obviously the most advanced and should grab your attention over the other.



Rotate replica sets

Rotating a timestamped set of replicas

This is a very basic approach, it consists in storing your replicas in a timestamped folder and then rotating them based on number of replica sets, date, or size of the target folder.

This technique is easy to implement and simple to understand. It only has one drawback: as the timestamped folder will always be different, the whole VM will be copied every time.

This can be convenient in case of small VMs (up to 100GB) stored in fast servers, but you will soon hit reality if you want to apply it to multiterabyte VM sets.

Example:

./xsibackup --replica "VMs(WINSRV2019)" "root@192.168.33.200:22:/volume1/backup/replicas/$(date +%Y%m%d%H%M%S)" --rotate=7

Still, the results are quite convenient, as every VM in any target dir is a replica ready to be used.

Rotating a circular set of replicas with or without CBT (recommended)

This is a much more advanced technique that consists in making the target folder traverse a fixed set of folders in a round robin fashion. We can easily achieve that by using the MOD operator per instance.

./xsibackup --replica=cbt "VMs(WINSRV2019)" "root@192.168.33.200:22:/volume1/backup/replicas/$(( $(date +%j) % 5 ))"
-- OR --
(*) ./xsibackup --replica=cbt "VMs(WINSRV2019)" "root@192.168.33.200:22:/volume1/backup/replicas/$(( $(( $(date +%s)/86400 )) % 5 ))"

(*) Please note that some versions of ©ESXi lack the %j modifier in the date command, thus we offer an alternative method which consists in calculating the modulus over the number of days since Jan 1st 1970. Note that by dividing the Unix Timestamp by 86400 we get an integer (no floating point in bash arithmetics).

The date command $(date +%j) returns the Julian day (ordinal number for the day in the current year), then we obtain the modulus for the number of sets we want to keep (5 in the example). This way the target folder will subsequently be:

/volume1/backup/replicas/0
/volume1/backup/replicas/1
/volume1/backup/replicas/2
/volume1/backup/replicas/3
/volume1/backup/replicas/4

Some ESXi builds lack the %j modifier in the date command, you will notice that because you will receive an -sh: arithmetic syntax error. You can easily use the alternative code below, which uses the %s (Unit Timestamp) modifier, dividing by the number of seconds in the day (84600) will always return an integer, as bash arithmetics are not floating point.

./xsibackup --replica=cbt "VMs(WINSRV2019)" "root@192.168.33.200:22:/volume1/backup/replicas/$(( $(date +%s)/86400%5 ))"

Random rotation

Instead of rotating based on the day, you can make your replicas rotate based on the second. This will return a randomized set of folders in a range defined by the MODULUS operator as in the examples above.

./xsibackup --replica=cbt "VMs(WINSRV2019)" "root@192.168.33.200:22:/volume1/backup/replicas/$(( $(date +%s) % 5 ))"


The above will produce 5 folders (0-4) as in the previous examples, the only difference is that the sequence number will be random. This is useful if you are rotating replicas within the same day.

You only need to divide the Unix Timestamp by 60 or 3600 to rotate to the minute or the hour.

./xsibackup --replica=cbt "VMs(WINSRV2019)" "root@192.168.33.200:22:/volume1/backup/replicas/$(( $(( $(date +%s)/60 )) % 5 ))" # per minute rotation

./xsibackup --replica=cbt "VMs(WINSRV2019)" "root@192.168.33.200:22:/volume1/backup/replicas/$(( $(( $(date +%s)/60/60 )) % 5 ))" # per hour rotation


Every newly created folder will take a full VM transfer, you can also just clone them in the target server if that is more convenient. If you don't activate the CBT feature, like in the case of a Pro version, the whole -flat.vmdk files will be traversed and only the changed blocks will be sent to the target folder.

If you can afford to buy a full ©XSIBackup-DC version, you can activate CBT as in the example. This will cause the replication process to just send the changed blocks. Our CBT implementation allows to keep track of multiple targets at the same time, thus you will just be sending the changed blocks to each of the target folders and saving a vast amount of time in case you are replicating big VMs.

Note how we have omitted the --rotate argument in the last example, as the rotation is implicit to the programmatic definition of the target folder.