XSIBackup-DC regexp exclude vmdks help

sdan · 2023-03-02 08:51:56

I have vms which have some of their vmdks files are controlled by a kubernets cluster.

I want to exclude from backup all these vmdks files.

These vmdk files are located only in the following 2 folders:
/vmfs/volumes/datastorekubedata/kubevols
/vmfs/volumes/datastorekubedata/kubestatic

What regexp expression I can use with the --exclude option?

I do not want to backup any of these vmdk files because the kubernets clusters is behaving strange when the backup job is running.

Thank you

admin · 2023-03-03 09:06:57

What about:

*.datastorekubedata/kubevols*.
*.datastorekubedata/kubestatic*.

Or

^/vmfs/volumes/datastorekubedata/kubevols*.
^/vmfs/volumes/datastorekubedata/kubestatic*.

sdan · 2023-03-03 12:46:53

I run this command:
/vmfs/volumes/datastorelocal5/XSIBackup-DC/xsibackup \
--replica \
"VMs(testvm)" \
"/vmfs/volumes/datastoreringiernfs/esxivm" \
--disable-vmotion="" \
--use-smtp="1" \
--mail-to="sdan@globalis.ro" \
--exclude="((*.datastorekubedata/kubevols*.)|(*.datastorekubedata/kubestatic*.))" \

I get this errors:
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.6.0.6: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
Shell binary pool memory size: 2048 MB
-------------------------------------------------------------------------------------------------------------
Shell memory pool size was set to 2048 MB by default
-------------------------------------------------------------------------------------------------------------
Total available memory: 249319 MB (~1GB is always kept as a security margin)
-------------------------------------------------------------------------------------------------------------
License: 000406F100000000000000000017a477083a | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
PID: 11888439, Running job as: root
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/datastoreringiernfs/esxivm
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
VMotion interface <vmk0> has been disabled
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/datastoreringiernfs/esxivm'
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: 37083119108096 (34.54 TB)
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
testvm Hardware Version is: 14
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: testvm
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM: testvm (powered on)...
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: testvm
-------------------------------------------------------------------------------------------------------------
Backup start date: 2023-03-03T12:32:14
-------------------------------------------------------------------------------------------------------------
The size of the data to backup is huge, more memory will be allocated
-------------------------------------------------------------------------------------------------------------
Memory size was automatically set to: 1912 MB
-------------------------------------------------------------------------------------------------------------
2023-03-03 12:32:14 | Backing up 135 files, total size is 4.90 TB
-------------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-------------------------------------------------------------------------------------------------------------
Failed to initialize regex struct: string /vmfs/volumes/590a00bc-5aef1955-ec07-0017a4770810/testvm/testvm-3d06bdf4.hlog, pattern ((*.datastorekubedata/kubevols*.)|(*.datastorekubedata/kubestatic*.)), error: Cannot allocate memory
1/135 testvm-3d06bdf4.hlog 91.00 B | Done 0.00%
-------------------------------------------------------------------------------------------------------------
Failed to initialize regex struct: string /vmfs/volumes/590a00bc-5aef1955-ec07-0017a4770810/testvm/testvm.vmx, pattern ((*.datastorekubedata/kubevols*.)|(*.datastorekubedata/kubestatic*.)), error: No such file or directory
2/135 testvm.vmx 3.53 KB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
Failed to initialize regex struct: string /vmfs/volumes/590a00bc-5aef1955-ec07-0017a4770810/testvm/testvm-flat.vmdk, pattern ((*.datastorekubedata/kubevols*.)|(*.datastorekubedata/kubestatic*.)), error: No such file or directory
3/135 testvm-flat.vmdk 16.00 GB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 55.73% done | block 9130 out of 16384 | 221 MB/s | Done 0.18%
-------------------------------------------------------------------------------------------------------------
| 2023-03-03T12:32:57 | Error code 222 at file signal.c, line 222 | Error description: |

This vm has 3 harddisk:
[root@esxi5:/vmfs/volumes/5d0b3eec-7da7d463-ed8e-0017a477083a/XSIBackup-DC] cat /vmfs/volumes/vmstorage/testvm/testvm.vmx|grep fileName
scsi0:0.fileName = "testvm-000001.vmdk"
scsi0:1.fileName = "/vmfs/volumes/5d092ea7-c0c36652-bbd2-0017a477082c/kubestatic/testvm_1-000001.vmdk"
scsi0:2.fileName = "/vmfs/volumes/5d092ea7-c0c36652-bbd2-0017a477082c/kubevols/kubernetes-dynamic-pvc-testvm1212-000001.vmdk"

When I tried the second regexp you suggested to me (it's not working at all):

It looks like is trying to backup all the vmdks from that 2 folders.
I get this:

[root@esxi5:/vmfs/volumes/5d0b3eec-7da7d463-ed8e-0017a477083a/XSIBackup-DC] /vmfs/volumes/datastorelocal5/XSIBackup-DC/xsibackup \
> --replica \
> "VMs(testvm)" \
> "/vmfs/volumes/datastoreringiernfs/esxivm" \
> --disable-vmotion="" \
> --use-smtp="1" \
> --mail-to="sdan@globalis.ro" \
> --exclude="((^/vmfs/volumes/datastorekubedata/kubevols*.)|(^/vmfs/volumes/datastorekubedata/kubestatic*.))"
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.6.0.6: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
Shell binary pool memory size: 1912 MB
-------------------------------------------------------------------------------------------------------------
Shell memory pool size was set to 2048 MB by default
-------------------------------------------------------------------------------------------------------------
Total available memory: 249322 MB (~1GB is always kept as a security margin)
-------------------------------------------------------------------------------------------------------------
License: 000406F100000000000000000017a477083a | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
PID: 11894061, Running job as: root
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/datastoreringiernfs/esxivm
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
VMotion interface <vmk0> has been disabled
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/datastoreringiernfs/esxivm'
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: 37083119108096 (34.54 TB)
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
testvm Hardware Version is: 14
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: testvm
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM: testvm (powered on)...
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: testvm
-------------------------------------------------------------------------------------------------------------
Backup start date: 2023-03-03T12:41:51
-------------------------------------------------------------------------------------------------------------
The size of the data to backup is huge, more memory will be allocated
-------------------------------------------------------------------------------------------------------------
Memory size was automatically set to: 1912 MB
-------------------------------------------------------------------------------------------------------------
2023-03-03 12:41:51 | Backing up 142 files, total size is 4.90 TB
-------------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-------------------------------------------------------------------------------------------------------------
1/142 testvm-3d06bdf4.hlog 91.00 B | Done 0.00%
-------------------------------------------------------------------------------------------------------------
2/142 testvm.vmx 3.53 KB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
3/142 testvm-flat.vmdk 16.00 GB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 16384 out of 16384 | 388 MB/s | Done 0.33%
-------------------------------------------------------------------------------------------------------------
4/142 testvm.vmdk 523.00 B | Done 0.33%
-------------------------------------------------------------------------------------------------------------
5/142 testvm.vmsd 1.38 KB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
6/142 testvm.vmsd.tmp 732.00 B | Done 0.33%
-------------------------------------------------------------------------------------------------------------
7/142 vmx-testvm-2082346798-2.vswp [skipped excluded]
-------------------------------------------------------------------------------------------------------------
8/142 testvm.vmx.tmp 3.41 KB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
9/142 vmware-1.log 312.57 KB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
10/142 vmware-2.log 265.77 KB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
11/142 testvm.nvram 8.48 KB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
12/142 testvm-7c1e172e.vswp [skipped excluded]
-------------------------------------------------------------------------------------------------------------
13/142 testvm.vmx.lck [skipped excluded]
-------------------------------------------------------------------------------------------------------------
14/142 vmware.log 331.37 KB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
15/142 testvm.vmx~ 3.53 KB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
16/142 testvm-aux.xml [skipped excluded]
-------------------------------------------------------------------------------------------------------------
17/142 testvm-Snapshot2.vmsn 20.14 KB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
18/142 testvm-000001-sesparse.vmdk 84.00 MB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
19/142 testvm-000001.vmdk 308.00 B | Done 0.33%
-------------------------------------------------------------------------------------------------------------
20/142 testvm-Snapshot3.vmsn 20.16 KB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
21/142 testvm-000002-sesparse.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
22/142 testvm-000002.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
23/142 integrator-mysql-bigquery-live-flat.vmdk [open excluded]
-------------------------------------------------------------------------------------------------------------
24/142 testvm_1-flat.vmdk 20.00 GB | Done 0.33%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 7.36% done | block 1508 out of 20480 | 79 MB/s | Done 0.36%
-------------------------------------------------------------------------------------------------------------
| 2023-03-03T12:43:01 | Error code 222 at file signal.c, line 222 | Error description: |
| Signal Handler: the process received signal 2 |
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
The execution was halted as per the user request, signal will be added to error list
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
| 2023-03-03T12:43:01 | Error code 251 at file signal.c, line 251 | Error description: |
| raised SIGTERM (2) in job, num of errors: 1 |
-------------------------------------------------------------------------------------------------------------
Removed current target dir: /vmfs/volumes/bba6dffc-da391fbc/esxivm/testvm
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
| 2023-03-03T12:43:02 | Error code 304 at file signal.c, line 304 | Error description: |
| total errors: 2, check error.log |
-------------------------------------------------------------------------------------------------------------
Cleaning up...
-------------------------------------------------------------------------------------------------------------
VMotion Interface <vmk0> was re-enabled
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
Removed PID OK
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
| 2023-03-03T12:43:02 | Error code 222 at file signal.c, line 222 | Error description: |
| Signal Handler: the process received signal 11 |
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
| 2023-03-03T12:43:02 | Error code 299 at file signal.c, line 299 | Error description: |
| SIGTERM (11) condition was trapped: check logs for more details, error:description: |
| No such file or directory |
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
| 2023-03-03T12:43:02 | Error code 304 at file signal.c, line 304 | Error description: |
| total errors: 5, check error.log |
-------------------------------------------------------------------------------------------------------------
Cleaning up...
-------------------------------------------------------------------------------------------------------------
VMotion Interface <vmk0> was re-enabled
-------------------------------------------------------------------------------------------------------------

admin · 2023-03-04 19:55:58

Your REGEXP pattern is being rejected. Why are you using double parentheses?
Check this post: [url=https://www.educative.io/answers/how-to-write-regular-expressions-in-c]How to write regular expressions[/url]

Use something like this:

--exclude="[^/vmfs/volumes/datastorekubedata/kubevols*.]"

sdan · 2023-03-06 06:49:07

Hello

There is something wrong with your backup logic.

Please read the output of your script:

This vm testvm has 3 disks (each disk 20GB). 60 GB in total
scsi0:0.fileName = "testvm-000001.vmdk"
scsi0:1.fileName = "/vmfs/volumes/5d092ea7-c0c36652-bbd2-0017a477082c/kubestatic/testvm_1-000001.vmdk"
scsi0:2.fileName = "/vmfs/volumes/5d092ea7-c0c36652-bbd2-0017a477082c/kubevols/kubernetes-dynamic-pvc-testvm1212-000001.vmdk"

but your program wants to backup for this vm 4TB of data, the entire content of these 2 folders kubestatic and kubevols, not only their own vmdk,

We just bought 4 licenses of your product and we cannot use it.

|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.6.0.6: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
Shell binary pool memory size: 1912 MB
-------------------------------------------------------------------------------------------------------------
Shell memory pool size was set to 2048 MB by default
-------------------------------------------------------------------------------------------------------------
Total available memory: 249259 MB (~1GB is always kept as a security margin)
-------------------------------------------------------------------------------------------------------------
License: 000406F100000000000000000017a477083a | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
PID: 12002704, Running job as: root
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/datastoreringiernfs/esxivm
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
VMotion interface <vmk0> has been disabled
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/datastoreringiernfs/esxivm'
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: 37043876790272 (34.50 TB)
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
testvm Hardware Version is: 14
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: testvm
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM: testvm (powered on)...
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: testvm
-------------------------------------------------------------------------------------------------------------
Backup start date: 2023-03-06T06:36:10
-------------------------------------------------------------------------------------------------------------
The size of the data to backup is huge, more memory will be allocated
-------------------------------------------------------------------------------------------------------------
Memory size was automatically set to: 1912 MB
-------------------------------------------------------------------------------------------------------------
2023-03-06 06:36:10 | Backing up 135 files, total size is 4.90 TB

admin · 2023-03-08 08:57:17

As any other software (c)XSIBackup has to make some assumptions. In this case the asumption is that .vmdk disks are stored in folders, generally one, but could be more, and that the rest of files in those folders are other constituent files of the VM, namely: .vmsd, vmxf, snapshots, etc..., which are indeed backed up. Should any other file not referenced by the .vmx file be found there, it will also be backed up by default.

Now the thing is: why are there in these folders other .vmdk files which don't belong to the VM?. Maybe this is the source of the problem. Still you can exclude individual files based on patterns. Send us the contents of your VM folder clearly marking the files that you don't want to backup and we'll create the REGEP for you.

sdan · 2023-03-08 09:23:23

Hello

These 2 folders contains all the vmdk managed directly by the kubernetes cluster. These vmdks belongs to other vm nodes controlled by the kubernetes cluster. It does not conaint any vmx , log, .vmfs, vmxf files...
I will feel more safe when I will make the backup to completely exclude this type of folders...

I managed to find the right regexp I need it. but
I still get to each vm I want to backup
"2023-03-06 06:36:10 | Backing up 135 files, total size is 4.90 TB"
but the files are excluded...

vmfs/volumes/datastorelocal5/XSIBackup-DC/xsibackup \
--replica \
"VMs(testvm)" \
"/vmfs/volumes/datastoreringiernfs/esxivm" \
--disable-vmotion="" \
--use-smtp="1" \
--mail-to="sdan@globalis.ro" \
--exclude="(kubestatic|kubevols)"

admin · 2023-03-08 13:30:28

OK, great, but, did you finally achieved what you wanted?.
It seems as if you were just being misguided by the fact that all files are listed, then the excluded ones are marked as 'excluded', that's normal behavior.

sdan · 2023-03-09 08:02:50

Yes. I managed to do it, but...

In the next versions can you print this message after the exclusion is happening?

"2023-03-06 06:36:10 | Backing up xxx files, total size is XXX TB"

Now is misleading.

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

#1 2023-03-02 08:51:56

XSIBackup-DC regexp exclude vmdks help

#2 2023-03-03 09:06:57

Re: XSIBackup-DC regexp exclude vmdks help

#3 2023-03-03 12:46:53

Re: XSIBackup-DC regexp exclude vmdks help

#4 2023-03-04 19:55:58

Re: XSIBackup-DC regexp exclude vmdks help

#5 2023-03-06 06:49:07

Re: XSIBackup-DC regexp exclude vmdks help

#6 2023-03-08 08:57:17

Re: XSIBackup-DC regexp exclude vmdks help

#7 2023-03-08 09:23:23

Re: XSIBackup-DC regexp exclude vmdks help

#8 2023-03-08 13:30:28

Re: XSIBackup-DC regexp exclude vmdks help

#9 2023-03-09 08:02:50

Re: XSIBackup-DC regexp exclude vmdks help

Board footer