Last updated on Monday 28th of February 2022 08:52:48 PM

©XSIBackup Extended Rotation

Advanced rotation features to build standalone data protection solutions

(*) Do not forget to start at Rotation documentation in the ©XSIBackup-DC manual.

Before you consider anything else

©XSIBackup-DC's rotation features are heavily dependent on the date time stamp

$(date +%Y%m%d%H%M%S)


It is simply a typical timestamp containing the date and time in a pseudo ISO time format (without any separator characters), namely: YYYYDDMMhhmmss. If you are a sysadmin you should be used to this kind of naming scheme and if you are an advanced user, it's about time that you do ;-)

The code you are seeing above is just the bash date command being used to generate that string. You will normally use that string, and not other, to automatically generate subfolders where to store backups or replicas. It is indeed the format that the deduplication engine uses to generate backup subfolders.

(*) You must observe this constraint or you won't get rotation working properly.

Rotate VM backup sets

General rotation principles

Rotation can happen at the beginning of the job at the VM level or at the end of it. When we leave some things until the end is to follow some basic conservation principle: "do not delete anything until you have at least completed copying the new thing". This is especially useful in case of users that have little backup space when compared to the virtual machines' size.

We have added some new features to allow to squeeze your available backup room even more, like the "S" option (commented below) which will allow to consider the VM size on disk instead of its full nominal size. This will become especially useful for those of you using full replicas as backups.

When using the "S" option ©XSIBackup will use a default File System block size of 512 bytes, which will work fine with VMFS and with most other underlying FSs that you may be using over NFS. You can change that with the --fsblocksize argument. You should not need to use it, in fact you should avoid using exotic configurations in sake of your data's integrity, still it's there just in case.

Different ways to rotate your backups & replicas

There are different approaches to rotate your data, you should choose the option that best fits your needs.

--rotate={N} (post): rotate a fixed number of folders {N}. Easy to configure if you have space enough to not have to worry about it.

--rotate={N}d (post): rotate a fixed number of days {N}. Once some backup folder becomes older than {N} days, it will be automatically deleted.

--rotate={N}[GB|TB] (pre): backup up until a fixed predefined amount of space {N} has been reached and then start deleting the eldest folders to make room.

--rotate=max (pre): this flag will start to delete the eldest folders in some volume once the free space is below 110% of the VM about to be backed up.

(*) Post & pre means wether the pruning process is performed at the end of the backup job or right before backing up each VM.

Other options relative to --rotate feature

--options=S (optional)(v. 1.5.1.7): the "S" option will make ©XSIBackup consider the non-zero data in a sparse virtual hard disk instead of the full nominal size of a disk at the time to evaluate the VM size.

--options=f (optional)(v. 1.5.1.7): force autoprovision even in the event that you have chosen a rotation scheme based on number instead of size.

--fsblocksize={N} (optional)(v. 1.5.1.7): it takes an integer value representing bytes. You should never use this option, unless you are a very advanced user and you know very well what you are doing. You can easily cut yourself with this razor, it's extremely sharp.

Differences in rotating --backup or --replica sets

Backup sets are generated using the above timestamp, when you add the --rotate argument, whether with a single figure to rotate by number of backups, by volume of data or by days, the rotation process will know where to perform that rotation. It will somehow be obvious, as the backup repositories are already generated with a defined structure that allows to easily know where each thing is.

When you use the --replica action to generate directly usable copies of VMs, you are allowed to place this replicas at any point in a local or remote file system. Thus, it will not be that obvious to the rotation process where it has to count accumulated data to know whether it was exceeded in order to delete old folders. It uses some guesses though, which are useful and work well most of the times.

It checks to see if the last dir where the replica is being placed is using the above time stamp convention. If so, it will use the directory immediately below as the root of the rotation. Thus the older time stamped folders in that directory will be deleted by the rotation process and the volume of data accumulated in that root dir will be used to compare against the predefined limits.

--rotate-at

This is yet another argument associated to the rotation process. This is the root folder where ©XSIBackup-DC will consider contents to be stored at. Thus, when you use some user defined space, i.e.: --rotate=500GB, it will check the size of the contents of this folder.

You can use the --rotate-at argument, to which you can pass any local or remote path. When any of the predefined limits are reached, the rotation process will delete the eldest folders in that root path.

When a timestamped folder in the form YYYYMMDDhhmmss is detected at the end of the target argument ©XSIBackup-DC will automatically consider the previous directory to be the root of the rotation, nonetheless you can tweak that to your needs.

When does pruning happen when you employ --rotate

There are two different behaviours: when you rotate per number or per days, the pruning process is performed at the end of the backup, whereas when you rotate by volume of data the pruning is performed right before each VM backup, should some space need to be freed. In case you use --options=f (force provision) and you have run out space, the deltion of old folders/ backups will be done before backing up any VM.

You might see that the resulting set of VM replicas or backups contains some extra folder when you rotate by number, as the backup/ replica folder being currently copied will not be considered.

Whatch out before you go

Size rotation feature requires as much free space as 110% of the full VM nominal size, namely 10% more space than the nominal size of the VM. It doesn't matter if the disk is sparse and only a part of the space is used. Thus, if your VM has two 100GB disks with only half of the space used, you will need 220GB free to backup that VM. It won't use 220GB though, just 100GB (the non-zero data). This implies that in the end you have to take this fact into account when reserving space.

Should you want to create a set of replicas for some VM which is 200GB in size, you will need at least 440GB to store them, otherwise the rotation process will delete all previous replicas every time. In the end you will be able to keep 2 replicas in those 440GB. If you enlarge the backup user space to 540GB (--rotate=540GB) you will be able to keep 3 replicas, but you will never use more than 300GB of real space, even though your set limit will be 540GB.

-- Update 2022-01-09 --
You can clear the last constraint by using the "S" option (v. 1.5.1.7). Still this requires that you are aware of the limitations of this option. VMFS can hold sparse files, still the behaviour they may present will in turn depend on the kind of virtual disk you are using and the kind of backup volume. As a rule of thumb, if you are using thin provisioning on VMFS, which you should be doing as per 2022, and your backup volume is a FS that can hold sparse files, you will be able to use it flawlessly. If you don't know or if you are unsure, stick to default behaviour.

This applies of course to sparse files, which is the default virtual disk format that ©XSIBackup-DC creates. Should you use some volume that doesn't allow sparse files, you would need to consider the full VM size as non-zero data, it's weird nowadays though, most of the FSs you may use are indeed able to handle sparse files.

What it means in the end is that you will always need to leave a margin equal to the full size of the VM plus an additional 10% and that used space calculation will be made without taking compression into account. Our space calculations when rotating based on space figures are conservative to the extreme on purpose. If you have a backup volume and you want to fill it up, good luck (you will need it), try with a fixed rotation strategy or with a fixed number of days, but don't complain if you end up hurting yourself.

This applies as well to backup repositories, although in this case the size of the backup will be drastically reduced due to deduplication plus eventual compression (default) on the backup repository. First time you backup a VM to a repository, you will need roughly speaking 50-60% of its non-zero data volume to store it, still you will be asked to provide 110% of its nominal size. On subsequent backups just the differential data volume will be required, still you will need 110% of the VM nominal size to be available.

-- Update 2022-01-09 --
This is a good practice when you are using SSD/NVMe/M.2 disks, as they will require that you leave some 10% free (at a minimum) to operate well. This limitations will probably be waived as new technologies emerge. In any case trying to use the last GBs of your disks is something among the worst and most dangerous things that somebody can do.

Note that this space requirement is applied on a per VM basis. Thus, you don't need 110% of the whole volume of all the virtual disks in a Virtual Machine set to be able to back it up, just 110% of the biggest one at the time to back it up.

The eventual need for space is recalculated previous to backing up each new virtual machine. Only if at that moment some lack of space is detected, will the rotate process be triggered to make it. Nonetheless the deletion of objects is performed at a timestamped folder level and that timestamped folder may contain multiple virtual machines.

You need a margin equal in size to 110% of your VM nominal size. If you can't afford to have that margin then use some other kind of rotation.