Anomaly between 'Needed room' and 'Available room'?

Random Thoughts · 2019-01-04 11:09:22

Hi all,

I've restricted backup room to 500GB, and 'Needed room' is 412GB. I've run a number of successful backups via a nightly cron job, but this is what I'm now seeing in my emails -

Available room in device /vmfs/volumes/NAS before backup: 404 Gb.
Sparse size on disk of the selected virtual machines: 412 Gb.
Needed room in device /vmfs/volumes/NAS for backup: 412 Gb.
Available space in device /vmfs/volumes/NAS after backup: 391 Gb.

What's happening here? My assumption was that when 'Available room' was less than 'Needed room', the earliest backup would be deleted to free up space.

Addendum: I've just checked the logs and found this -

Not enough room to make the backup, some older folders will be deleted

They haven't been deleted, presumably (I'll have to check this) because the NFS share doesn't currently allow file deletion. So am I right in thinking that XSIbackup will continue to backup VMs while it can (disk space permitting)?

Edit: Just tried deleting the earliest backup folder manually (from SSH connection on VMWare) and it completed without any issues so I'm not sure why XSIBackup wasn't able to delete it

Thanks for looking

Last edited by Random Thoughts (2019-01-04 11:39:09)

Random Thoughts · 2019-01-04 17:15:46

Done hot backup (id: 001) using vmkfstools (no compression)
Backing up to folder /vmfs/volumes/NAS/20190104020004
The backup room has been limited to 500 Gb.
Available room in device /vmfs/volumes/NAS before backup: 404 Gb.
Sparse size on disk of the selected virtual machines: 412 Gb.
Needed room in device /vmfs/volumes/NAS for backup: 412 Gb.
(Id)VM Name 	State 	Size (Gb) 	Stop 	Copy 	Start 	Time (min) 	Speed (mb/s)
(10) Hawking	ON	204/ 204	NO (hot backup)	OK	- 	35 	97/ 97
(11) Kepler	ON	200/ 200	NO (hot backup)	OK	- 	37 	90/ 90
(12) Hubble	ON	8/ 8	NO (hot backup)	OK	- 	22 	5/ 5
Available space in device /vmfs/volumes/NAS after backup: 391 Gb.
Complete backup elapsed time: 96 min

"/vmfs/volumes/datastore1/xsi-dir/xsibackup" \
--backup-point=/vmfs/volumes/NAS \
--backup-type=all \
--date-dir=yes \
--backup-room=500 \
--snapshot=doquiesce \
--backup-id=001 \
--description="Nightly backup" \
--mail-to=danny@trisect.uk \
--use-smtp=1 \
--exec=yes >> "/vmfs/volumes/datastore1/xsi-dir/var/logs/xsibackup.log"[root@edge:/vmfs/volumes/5

admin · 2019-01-06 16:05:27

Oops, sorry, we messed up with a different thread.
Right, you are not using XSITools, so our answers are useless.

The regular space provissioning mechanism deletes files which are named according to the [b]datedirmask[/b], i.e.: [b]20190104034551[/b] variable in the [b]conf/xsiopts[/b] file, it will ignore any other folder and it will only delete folders in the [b]--backup-point[/b] root.

If you don't allow deletion, then your disk will fill up.

Random Thoughts · 2019-01-06 19:35:25

Thanks for clearing up the confusion between threads,

This is the current setting in conf/xsiopts -

datedirmask="20[1-3][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]"

These are the remote directories created by XSIbackup -

root@locker:/shared/vmware# ls -al
total 40
drwxrwsrwx+ 10 root   users 4096 Jan  6 02:02 .
drwxr-xr-x   4 root   root  4096 Jan  2 15:32 ..
drwxr-xr-x+  5 nobody users 4096 Dec 30 03:18 20181230020004
drwxr-xr-x+  5 nobody users 4096 Dec 31 03:10 20181231020004
drwxr-xr-x+  5 nobody users 4096 Jan  1 03:14 20190101020003
drwxr-xr-x+  5 nobody users 4096 Jan  2 03:16 20190102020003
drwxr-xr-x+  5 nobody users 4096 Jan  3 03:21 20190103020003
drwxr-xr-x+  5 nobody users 4096 Jan  4 03:15 20190104020004
drwxr-xr-x+  5 nobody users 4096 Jan  5 03:12 20190105020004
drwxr-xr-x+  5 nobody users 4096 Jan  6 03:15 20190106020004

As far as I can tell the datedirmask matches the created folders, I'll let it run for a few days and see what happens, unless you can see an obvious problem with what I've posted

admin · 2019-01-06 20:40:32

That looks fine, once you reach the limit set in the [b]datedirmask[/b] variable, if the sum of all the folders matching the mask exceeds the value set in [b]--backup-room[/b] argument, the folders will start to be deleted starting by the eldest one. They will be removed until making enough room to fit the current backup.

You should also take a look at the [b][https://33hops.com/xsibackup-help-man-page.html#deldirs]--del-dirs[/url][/b] argument.

Random Thoughts · 2019-01-06 20:52:48

[quote=admin]if the sum of all the folders matching the mask exceeds the value set in [b]--backup-room[/b] argument, the folders will start to be deleted starting by the eldest one. They will be removed until making enough room to fit the current backup.
[/quote]

If this is the case I've misunderstood --backup-room, and the information given in emails. I'm trying a little experiment by setting --backup-room=50, each backup is roughly 14GB in size, so with the current state of the backup folder it should delete the earliest backups.

I'll post the outcome sometime tomorrow

p.s. the --del-dirs option only seems to be available in the Pro version

admin · 2019-01-06 21:08:40

[b]--backup-room[/b] sets the maximum size you wish your set of backups to use. Of course there are operational margins, so you should not expect the mechanism to work at the Mb. precission.

You can inspect the code beginning by the [b]make_room[/b] function in the [b]xsibackup[/b] file to check out how it works.

Random Thoughts · 2019-01-07 10:30:40

Ok, first backup done with [b]--backup-room=50[/b], and it's......interesting!!

The backup room has been limited to 50 Gb.
Available room in device /vmfs/volumes/NAS before backup: -59 Gb.
Sparse size on disk of the selected virtual machines: 412 Gb.
Needed room in device /vmfs/volumes/NAS for backup: 412 Gb.
(Id)VM Name 	State 	Size (Gb) 	Stop 	Copy 	Start 	Time (min) 	Speed (mb/s)
(10) Hawking	ON	204/ 204	NO (hot backup)	OK	- 	46 	75/ 75
The eldest folders were deleted to make room:
Error MKROOM01: cannot make 216G of room, only 50G can be made available
/vmfs/volumes/NAS/20181230020004
/vmfs/volumes/NAS/20181231020004
/vmfs/volumes/NAS/2 0190101020003
/vmfs/volumes/NAS/20190102020003
/vmfs/volumes/NAS/20190103020003
/vmfs/volumes/NAS/20190104020004
/vmfs/volumes/NAS/20190105020004
/vmfs/volumes/NAS/20190106020004
(11) Kepler	ON	200/ 200	NO (hot backup)	OK	- 	42 	80/ 80
The eldest folders were deleted to make room:
Error MKROOM01: cannot make 212G of room, only 50G can be made available
/vmfs/volumes/NAS/20190107020003
(12) Hubble	ON	8/ 8	NO (hot backup)	OK	- 	22 	5/ 5
Available space in device /vmfs/volumes/NAS after backup: 41 Gb.
Complete backup elapsed time: 111 min
Backup of ESXi configuration is not available in XSIBACKUP-FREE
Get XSIBACKUP-PRO at https://33hops.com
USE DISCOUNT COUPON XSIDREWYI
• [ Mon Jan 7 02:08:45 UTC 2019 ] ERROR (MKROOM01), details Error: cannot make 216G of room, only 50G can be made available

• [ Mon Jan 7 02:48:27 UTC 2019 ] ERROR (MKROOM01), details Error: cannot make 212G of room, only 50G can be made available

What I read from this is that it was able to make room to back up Hawking by deleting older folders (which it did). It then had to delete this fresh folder containing the Hawking backup to make room for the Kepler & Hubble backups.

I've now set [b]--backup-room=250[/b] and cron to run the backup every three hours to give me a quicker understanding of the capacity management process.

Thank you admin (whoever you are) for helping me understand the process, I guess this thread can be marked as 'solved', though I will continue to tinker for a while to find a good balance between number of backups held and storage requirements.

admin · 2019-01-07 11:13:35

As you have probably noticed so far, we are not much commercially oriented, which is a problem, as geeky nerds also eat something from time to time.

Haven't you noticed that the value of your own time trying to fit your backups in your backup space exceeds by far the price of a Pro license, which features deduplication and would wipe all your problems away.

Random Thoughts · 2019-01-09 17:15:57

Ok you've convinced me!!

Now running XSIbackup-Pro 11.2.2, I think it's going to take me a little while to go through all the new options and find a 'best fit' for my needs

admin · 2019-01-09 17:56:04

Thank you!, If they find out they will move me to the commercial area ;-)
Just write to support with your tech request and we'll propose you the best fit

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

#1 2019-01-04 11:09:22

Anomaly between 'Needed room' and 'Available room'?

#2 2019-01-04 17:15:46

Re: Anomaly between 'Needed room' and 'Available room'?

#3 2019-01-06 16:05:27

Re: Anomaly between 'Needed room' and 'Available room'?

#4 2019-01-06 19:35:25

Re: Anomaly between 'Needed room' and 'Available room'?

#5 2019-01-06 20:40:32

Re: Anomaly between 'Needed room' and 'Available room'?

#6 2019-01-06 20:52:48

Re: Anomaly between 'Needed room' and 'Available room'?

#7 2019-01-06 21:08:40

Re: Anomaly between 'Needed room' and 'Available room'?

#8 2019-01-07 10:30:40

Re: Anomaly between 'Needed room' and 'Available room'?

#9 2019-01-07 11:13:35

Re: Anomaly between 'Needed room' and 'Available room'?

#10 2019-01-09 17:15:57

Re: Anomaly between 'Needed room' and 'Available room'?

#11 2019-01-09 17:56:04

Re: Anomaly between 'Needed room' and 'Available room'?

Board footer