Ignores backup-room parameter via rsync backup

ThomasA · 2018-09-27 06:15:45

Hello,

I use a rsync backup to another esxi host.
I defined the backup-room, and in the logs you can see that the parameter should work:

"The backup room has been limited to 400 gb."

But the backup are written till the disk is full.

Should the parameter work with rsync or are there any restrictions?

My CronJob:
"/vmfs/volumes/datastore1/xsi-dir/xsibackup" \
--backup-point=IP:/vmfs/volumes/DATASTORE-ID/backupESX1 \
--backup-type=running \
--mail-from=Email \
--mail-to=Email \
--smtp-srv=Mailserver \
--smtp-port=25 \
--backup-prog=rsync \
--test-mode=false \
--smtp-usr=x \
--smtp-pwd=x \
--smtp-auth=none \
--backup-vms=VMs List \
--backup-type=custom \
--subject=BACKUP-ESX1 \
--backup-id=001 \
--date-dir=yes \
--backup-room=400 \
--exec=yes >> "/vmfs/volumes/datastore1/xsi-dir/var/logs/xsibackup.log"

Thank you
Thomas

admin · 2018-10-03 08:32:48

We will revise this and eventually release a fix.
.....
....
...
This was reviewed and no problems were found, please keep on reading to know about how backup folders are considered for automatic deletion.

markus · 2018-11-02 09:36:04

Hi,

I can see the same error with xsitools backup to a NFS backup target.

I limited backup space to 4096 GB. The limit is shown in backup logs. But it does not free up needed space. In the mean time it used the full NFS storage and no further backup could be run successfully.

I'm using latest Pro version. Please find some parts of the log file below.

How can I clean up backup space and get back to working backup?

Best regards
Markus

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
XSIBackup PID: 36045737 XXXXXXXXX.local
Fri, 02 Nov 2018 09:22:29 +0000 IPv4: XXXXXXXXX/255.255.255.0
VMware ESXi 6.5.0 build-8935087 (c) Rsync 3.1.0 as opt. dependency
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Backup Id: 001 Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Backup description:XXXXXX | XSITools backup of production VMs
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Message: crontab is installed for user root
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
ADVICE: no SSD disks, please consider adding an SSD cache disk to improve performance
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Backup user is: root
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Backup program is: xsitools
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
/vmfs/volumes/qnap/xsi-backup is an XSITools repo
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
sh: bad number
--backup-point found at /vmfs/volumes/qnap/xsi-backup (filesystem: NFS)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
WARNING: the </vmfs/volumes/qnap/xsi-backup> dir is not an XSITOOLS repo, it will be initialized
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
The backup room has been limited to 4096 gb.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Getting list of all VMs...
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 win10-vpc [vmfs01] win10-vpc/win10-vpc.vmx windows9_64Guest vmx-13
12 XXXX-ustrich [vmfs01] XXXX-ustrich/XXXX-ustrich.vmx windows9Server64Guest vmx-11 based on Dell Systembuilder ISO
3 XXXX-PROXY [vmfs01] XXXX-PROXY/XXXX-PROXY.vmx other26xLinuxGuest vmx-07
4 XXXX-CLOUD [vmfs01] XXXX-CLOUD/XXXX-cloud.vmx otherLinux64Guest vmx-07
7 XXXX-SBS01 [vmfs01] XXXX-SBS01/XXXX-SBS01.vmx windows7Server64Guest vmx-07
8 XXXX-VPN [vmfs01] XXXX-VPN/XXXX-VPN.vmx debian8_64Guest vmx-13
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
VMs to backup:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 win10-vpc [vmfs01] win10-vpc/win10-vpc.vmx windows9_64Guest vmx-13
12 XXXX-ustrich [vmfs01] XXXX-ustrich/XXXX-ustrich.vmx windows9Server64Guest vmx-11 based on Dell Systembuilder ISO
3 XXXX-PROXY [vmfs01] XXXX-PROXY/XXXX-PROXY.vmx other26xLinuxGuest vmx-07
4 XXXX-CLOUD [vmfs01] XXXX-CLOUD/XXXX-cloud.vmx otherLinux64Guest vmx-07
7 XXXX-SBS01 [vmfs01] XXXX-SBS01/XXXX-SBS01.vmx windows7Server64Guest vmx-07
8 XXXX-VPN [vmfs01] XXXX-VPN/XXXX-VPN.vmx debian8_64Guest vmx-13
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Calculating sizes...
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Needed room: 1541 Gb.
Sparse size: 2185 Gb.
Available room: 0 Gb.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Not enough room to make the backup, some older folders will be deleted
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] Starting backup (size is 22653M on 102823M file)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
XSIBackup will backup your VMs while they are running and will quiesce guest services too, so that users
can continue to use the VM while the backup is taking place. You can also run cold and warm --backup-how
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Hot backup selected for VM: [win10-vpc], will not be switched off
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] info: boot partition is MBR
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] info: Win64 OS (windows9_64) detected on MBR, applying Windows algorithm
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] Info: attempting to free 34941 mb at [ /vmfs/volumes/qnap ]
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Removing snapshots, please wait...
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Syncronizing config files
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] warning: waiting for the backup dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] to be created. Consider disabling async NFS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
[win10-vpc] error: the dir [/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc] cannot be created, giving up. Details: mkdir: can't create directory '/vmfs/volumes/qnap/xsi-backup/20181102092229/': No space left on device
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
./xsibackup: line 1: can't create /vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc/win10-vpc.vmx: nonexistent directory
./xsibackup: line 1: can't create /vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc/win10-vpc.vmsd: nonexistent directory
cp: can't create '/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc': No such file or directory
cp: can't create '/vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc': No such file or directory
[win10-vpc] info: VMWare Tools detected, taking snapshot QUIESCED(false)...
[win10-vpc] info: set argument --snapshot=doquiesce to quiesce your VMs
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Backing up virtual disks...
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
DISK=/vmfs/volumes/vmfs01/win10-vpc/win10-vpc-000001-sesparse.vmdk
DISK=/vmfs/volumes/vmfs01/win10-vpc/win10-vpc-000001.vmdk
DISK=/vmfs/volumes/vmfs01/win10-vpc/win10-vpc-Snapshot63.vmsn
DISK=/vmfs/volumes/vmfs01/win10-vpc/win10-vpc.vmdk
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
grep: /vmfs/volumes/vmfs01/win10-vpc/win10-vpc-000001-sesparse.vmdk: Device or resource busy
(c) XSIDiff 1.1.1.4 licensed:
A full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/vmfs01/win10-vpc/win10-vpc-000001.vmdk
Error opening file on createfile() function: /vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc/win10-vpc-000001.vmdk, error: No such file or directory
Error creating file2: -1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
(c) XSIDiff 1.1.1.4 licensed:
A full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/vmfs01/win10-vpc/win10-vpc-Snapshot63.vmsn
Error opening file on createfile() function: /vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc/win10-vpc-Snapshot63.vmsn, error: No such file or directory
Error creating file2: -1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
(c) XSIDiff 1.1.1.4 licensed:
A full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/vmfs01/win10-vpc/win10-vpc.vmdk
Error opening file on createfile() function: /vmfs/volumes/qnap/xsi-backup/20181102092229/win10-vpc/win10-vpc.vmdk, error: No such file or directory
Error creating file2: -1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Blocksize is 52428800 bytes
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Compression has been set
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Backing up via XSITools
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
File name: win10-vpc-flat.vmdk, File size: 107374182400
Block size: 52428800, Block count: 2048
Disk usage: 23776460800
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Cannot process block 5422290059ed7a50fa42ecc25847aef3dfa07f4c (13), error: cat: write error: No space left
Retrying block 13 for the 1 time
Cannot process block 5422290059ed7a50fa42ecc25847aef3dfa07f4c (13), error: cat: write error: No space left
Retrying block 13 for the 2 time
13/2048 blocks | Processed 0%

admin · 2018-11-02 13:17:31

XSIBackup will only make room by deleting folders in the root dir which follow the pattern expressed by the datedirmask variable defined in conf/xsiopts

Which is the same that the --date-dir=yes argument uses BTW.

datedirmask="20[1-3][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]"

If you have some other data there that does not follow the mask in the root dir, XSIBackup will ignore it. This is by design, XSIBackup cannot happily delete data about which knows nothing.

markus · 2018-11-05 09:34:35

My directory structure looks like this:

[root@XXXXXXX:/vmfs/volumes/902e9598-8cb1d992/xsi-backup] ls -lah
total 96
drwxr-xr-x 24 root root 4.0K Nov 5 09:08 .
drwxrwxrwx 6 root root 4.0K Sep 3 08:21 ..
-rw-r--r-- 1 root root 0 Oct 4 19:24 .xsitools
drwxr-xr-x 8 root root 4.0K Sep 17 22:06 20180917180004
drwxr-xr-x 8 root root 4.0K Sep 18 22:06 20180918180003
drwxr-xr-x 8 root root 4.0K Sep 19 22:09 20180919180004
drwxr-xr-x 8 root root 4.0K Sep 20 22:06 20180920180004
drwxr-xr-x 8 root root 4.0K Sep 21 22:05 20180921180003
drwxr-xr-x 8 root root 4.0K Sep 22 21:40 20180922180004
drwxr-xr-x 8 root root 4.0K Sep 23 21:42 20180923180004
drwxr-xr-x 3 root root 4.0K Sep 24 14:23 20180924142303
drwxr-xr-x 8 root root 4.0K Sep 25 05:08 20180924180005
drwxr-xr-x 8 root root 4.0K Sep 26 04:49 20180925180005
drwxr-xr-x 8 root root 4.0K Sep 27 04:55 20180926180005
drwxr-xr-x 8 root root 4.0K Sep 28 04:53 20180927180005
drwxr-xr-x 8 root root 4.0K Sep 29 05:07 20180928180005
drwxr-xr-x 8 root root 4.0K Sep 30 04:49 20180929180005
drwxr-xr-x 8 root root 4.0K Oct 1 04:57 20180930180005
drwxr-xr-x 8 root root 4.0K Oct 2 04:54 20181001180005
drwxr-xr-x 8 root root 4.0K Oct 3 04:46 20181002180005
drwxr-xr-x 8 root root 4.0K Oct 4 05:10 20181003180005
drwxr-xr-x 7 root root 4.0K Oct 4 21:23 20181004180005
drwxr-xr-x 3 root root 4.0K Nov 2 09:10 20181102090930
drwxr-xr-x 18 root root 4.0K Aug 24 12:24 data
drwxr-xr-x 2 root root 4.0K Nov 3 15:31 logs

The naming convention was not chosen by me. It is the default of xsitools backup method.
All individual data blocks are saved in "data" directory.

[root@XXXXX:/vmfs/volumes/902e9598-8cb1d992/xsi-backup] du -csh *
2.3M 20180917180004
2.3M 20180918180003
2.3M 20180919180004
2.3M 20180920180004
2.3M 20180921180003
2.3M 20180922180004
2.3M 20180923180004
92.0K 20180924142303
2.3M 20180924180005
2.3M 20180925180005
2.3M 20180926180005
2.3M 20180927180005
2.3M 20180928180005
2.3M 20180929180005
2.3M 20180930180005
2.3M 20181001180005
2.3M 20181002180005
2.3M 20181003180005
832.0K 20181004180005
80.0K 20181102090930
4.9T data
12.0K logs
4.9T total

Option "datedirmask" is set to
datedirmask="20[1-3][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]"

How can I do a cleanup of older backups with xsitools backup method?

admin · 2018-11-05 10:58:30

XSITools stores files in repositories, as you can see a repository is compounded of a data folder, an .xsitools descriptor file and some timestamped folders which have nothing to do with regular backup folders, they just store the hash maps and auxiliary VM files of every backup operation, but do not contain data, they are not backup folders, they are the intermediate differential data description.

The backup folder is the repository itself, but as you have stored the XSITools repository to the root of the datastore. The backup folder is the datastore, not the (c)XSITools repository and has no name, but the upper level, which is the datastore and cannot be removed.

To have your XSITools repository considered for deletion, place it inside a folder in the root of the datastore, not in the root itself. You must make sure that the folder name follows the datedirmask convention. You can easily achieve that by using dynamic folders like:

To create a monthly XSITools repository that will be deleted when the disk is full.

--backup-point="/vmfs/volumes/backup/$(date +%Y%m'00000000')"

If you want to store quaterly.

--backup-point="/vmfs/volumes/backup3/$( date +"%Y %m" | awk '{Q=int($2/4)+1; printf("%sQ%s\n", $1, Q);}' )"

You have a more thorough explanation along with examples here:

https://33hops.com/xsibackup-pro-dynami … aming.html

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

#1 2018-09-27 06:15:45

Ignores backup-room parameter via rsync backup

#2 2018-10-03 08:32:48

Re: Ignores backup-room parameter via rsync backup

#3 2018-11-02 09:36:04

Re: Ignores backup-room parameter via rsync backup

#4 2018-11-02 13:17:31

Re: Ignores backup-room parameter via rsync backup

#5 2018-11-05 09:34:35

Re: Ignores backup-room parameter via rsync backup

#6 2018-11-05 10:58:30

Re: Ignores backup-room parameter via rsync backup

Board footer