©XSIBackup-Free: Free Backup Software for ©VMWare ©ESXi

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software


You are not logged in.

#1 2023-03-02 08:51:56

sdan
Member
Registered: 2023-03-02
Posts: 7

XSIBackup-DC regexp exclude vmdks help

I have vms which have some of their vmdks files are controlled by a kubernets cluster.

I want to exclude from backup all these vmdks files.

These vmdk files are located only in the following 2 folders:
/vmfs/volumes/datastorekubedata/kubevols
/vmfs/volumes/datastorekubedata/kubestatic

What regexp expression I can use with the --exclude option?

I do not want to backup any of these vmdk files because the kubernets clusters is behaving strange when the backup job is running.

Thank you

Offline

#2 2023-03-03 09:06:57

admin
Administrator
Registered: 2017-04-21
Posts: 2,057

Re: XSIBackup-DC regexp exclude vmdks help

What about:

*.datastorekubedata/kubevols*.
*.datastorekubedata/kubestatic*.

Or

^/vmfs/volumes/datastorekubedata/kubevols*.
^/vmfs/volumes/datastorekubedata/kubestatic*.

Offline

#3 2023-03-03 12:46:53

sdan
Member
Registered: 2023-03-02
Posts: 7

Re: XSIBackup-DC regexp exclude vmdks help

I run this command:
/vmfs/volumes/datastorelocal5/XSIBackup-DC/xsibackup \
--replica \
"VMs(testvm)" \
"/vmfs/volumes/datastoreringiernfs/esxivm" \
--disable-vmotion="" \
--use-smtp="1" \
--mail-to="sdan@globalis.ro" \
--exclude="((*.datastorekubedata/kubevols*.)|(*.datastorekubedata/kubestatic*.))" \

I get this errors:
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.6.0.6: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
Shell binary pool memory size: 2048 MB
-------------------------------------------------------------------------------------------------------------
Shell memory pool size was set to 2048 MB by default
-------------------------------------------------------------------------------------------------------------
Total available memory: 249319 MB (~1GB is always kept as a security margin)
-------------------------------------------------------------------------------------------------------------
License: 000406F100000000000000000017a477083a | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
PID: 11888439, Running job as: root
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/datastoreringiernfs/esxivm
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
VMotion interface <vmk0> has been disabled
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/datastoreringiernfs/esxivm'
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: 37083119108096 (34.54 TB)
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
testvm Hardware Version is: 14
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: testvm
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM: testvm (powered on)...
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: testvm
-------------------------------------------------------------------------------------------------------------
Backup start date: 2023-03-03T12:32:14
-------------------------------------------------------------------------------------------------------------
The size of the data to backup is huge, more memory will be allocated
-------------------------------------------------------------------------------------------------------------
Memory size was automatically set to: 1912 MB
-------------------------------------------------------------------------------------------------------------
2023-03-03 12:32:14 | Backing up 135 files, total size is 4.90 TB
-------------------------------------------------------------------------------------------------------------
    NUMBER                                                         FILE             SIZE          PROGRESS
-------------------------------------------------------------------------------------------------------------
Failed to initialize regex struct: string /vmfs/volumes/590a00bc-5aef1955-ec07-0017a4770810/testvm/testvm-3d06bdf4.hlog, pattern ((*.datastorekubedata/kubevols*.)|(*.datastorekubedata/kubestatic*.)), error: Cannot allocate memory
    1/135                                           testvm-3d06bdf4.hlog         91.00 B    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
Failed to initialize regex struct: string /vmfs/volumes/590a00bc-5aef1955-ec07-0017a4770810/testvm/testvm.vmx, pattern ((*.datastorekubedata/kubevols*.)|(*.datastorekubedata/kubestatic*.)), error: No such file or directory
    2/135                                                     testvm.vmx          3.53 KB   | Done   0.00%
-------------------------------------------------------------------------------------------------------------
Failed to initialize regex struct: string /vmfs/volumes/590a00bc-5aef1955-ec07-0017a4770810/testvm/testvm-flat.vmdk, pattern ((*.datastorekubedata/kubevols*.)|(*.datastorekubedata/kubestatic*.)), error: No such file or directory
    3/135                                               testvm-flat.vmdk         16.00 GB   | Done   0.00%
-------------------------------------------------------------------------------------------------------------
::: detail :::  55.73% done | block 9130 out of 16384 | 221 MB/s                            | Done   0.18%
-------------------------------------------------------------------------------------------------------------
|   2023-03-03T12:32:57 | Error code 222 at file signal.c, line 222 | Error description:                    |


This vm has 3 harddisk:
[root@esxi5:/vmfs/volumes/5d0b3eec-7da7d463-ed8e-0017a477083a/XSIBackup-DC] cat /vmfs/volumes/vmstorage/testvm/testvm.vmx|grep fileName
scsi0:0.fileName = "testvm-000001.vmdk"
scsi0:1.fileName = "/vmfs/volumes/5d092ea7-c0c36652-bbd2-0017a477082c/kubestatic/testvm_1-000001.vmdk"
scsi0:2.fileName = "/vmfs/volumes/5d092ea7-c0c36652-bbd2-0017a477082c/kubevols/kubernetes-dynamic-pvc-testvm1212-000001.vmdk"


When I tried the second regexp you suggested to me (it's not working at all):

It looks like is trying to backup all the vmdks from that 2 folders.
I get this:

[root@esxi5:/vmfs/volumes/5d0b3eec-7da7d463-ed8e-0017a477083a/XSIBackup-DC] /vmfs/volumes/datastorelocal5/XSIBackup-DC/xsibackup \
> --replica \
> "VMs(testvm)" \
> "/vmfs/volumes/datastoreringiernfs/esxivm" \
> --disable-vmotion="" \
> --use-smtp="1" \
> --mail-to="sdan@globalis.ro" \
> --exclude="((^/vmfs/volumes/datastorekubedata/kubevols*.)|(^/vmfs/volumes/datastorekubedata/kubestatic*.))"
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.6.0.6: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
Shell binary pool memory size: 1912 MB
-------------------------------------------------------------------------------------------------------------
Shell memory pool size was set to 2048 MB by default
-------------------------------------------------------------------------------------------------------------
Total available memory: 249322 MB (~1GB is always kept as a security margin)
-------------------------------------------------------------------------------------------------------------
License: 000406F100000000000000000017a477083a | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
PID: 11894061, Running job as: root
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/datastoreringiernfs/esxivm
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
VMotion interface <vmk0> has been disabled
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/datastoreringiernfs/esxivm'
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: 37083119108096 (34.54 TB)
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
testvm Hardware Version is: 14
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: testvm
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM: testvm (powered on)...
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: testvm
-------------------------------------------------------------------------------------------------------------
Backup start date: 2023-03-03T12:41:51
-------------------------------------------------------------------------------------------------------------
The size of the data to backup is huge, more memory will be allocated
-------------------------------------------------------------------------------------------------------------
Memory size was automatically set to: 1912 MB
-------------------------------------------------------------------------------------------------------------
2023-03-03 12:41:51 | Backing up 142 files, total size is 4.90 TB
-------------------------------------------------------------------------------------------------------------
    NUMBER                                                         FILE             SIZE          PROGRESS
-------------------------------------------------------------------------------------------------------------
    1/142                                           testvm-3d06bdf4.hlog         91.00 B    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    2/142                                                     testvm.vmx          3.53 KB   | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    3/142                                               testvm-flat.vmdk         16.00 GB   | Done   0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 16384 out of 16384 | 388 MB/s                           | Done   0.33%
-------------------------------------------------------------------------------------------------------------
    4/142                                                    testvm.vmdk        523.00 B    | Done   0.33%
-------------------------------------------------------------------------------------------------------------
    5/142                                                    testvm.vmsd          1.38 KB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
    6/142                                                testvm.vmsd.tmp        732.00 B    | Done   0.33%
-------------------------------------------------------------------------------------------------------------
    7/142                                  vmx-testvm-2082346798-2.vswp                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
    8/142                                                 testvm.vmx.tmp          3.41 KB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
    9/142                                                   vmware-1.log        312.57 KB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
   10/142                                                   vmware-2.log        265.77 KB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
   11/142                                                   testvm.nvram          8.48 KB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
   12/142                                          testvm-7c1e172e.vswp                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   13/142                                                testvm.vmx.lck                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   14/142                                                     vmware.log        331.37 KB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
   15/142                                                    testvm.vmx~          3.53 KB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
   16/142                                                testvm-aux.xml                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   17/142                                          testvm-Snapshot2.vmsn         20.14 KB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
   18/142                                    testvm-000001-sesparse.vmdk         84.00 MB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
   19/142                                             testvm-000001.vmdk        308.00 B    | Done   0.33%
-------------------------------------------------------------------------------------------------------------
   20/142                                          testvm-Snapshot3.vmsn         20.16 KB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
   21/142                                   testvm-000002-sesparse.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   22/142                                            testvm-000002.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   23/142                      integrator-mysql-bigquery-live-flat.vmdk                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   24/142                                             testvm_1-flat.vmdk         20.00 GB   | Done   0.33%
-------------------------------------------------------------------------------------------------------------
::: detail :::   7.36% done | block 1508 out of 20480 | 79 MB/s                             | Done   0.36%
-------------------------------------------------------------------------------------------------------------
|   2023-03-03T12:43:01 | Error code 222 at file signal.c, line 222 | Error description:                    |
|   Signal Handler: the process received signal 2                                                           |
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
The execution was halted as per the user request, signal will be added to error list
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
|   2023-03-03T12:43:01 | Error code 251 at file signal.c, line 251 | Error description:                    |
|   raised SIGTERM (2) in job, num of errors: 1                                                             |
-------------------------------------------------------------------------------------------------------------
Removed current target dir: /vmfs/volumes/bba6dffc-da391fbc/esxivm/testvm
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
|   2023-03-03T12:43:02 | Error code 304 at file signal.c, line 304 | Error description:                    |
|   total errors: 2, check error.log                                                                        |
-------------------------------------------------------------------------------------------------------------
Cleaning up...
-------------------------------------------------------------------------------------------------------------
VMotion Interface <vmk0> was re-enabled
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed PID                   OK
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
|   2023-03-03T12:43:02 | Error code 222 at file signal.c, line 222 | Error description:                    |
|   Signal Handler: the process received signal 11                                                          |
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
|   2023-03-03T12:43:02 | Error code 299 at file signal.c, line 299 | Error description:                    |
|   SIGTERM (11) condition was trapped: check logs for more details, error:description:                     |
|   No such file or directory                                                                               |
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
|   2023-03-03T12:43:02 | Error code 304 at file signal.c, line 304 | Error description:                    |
|   total errors: 5, check error.log                                                                        |
-------------------------------------------------------------------------------------------------------------
Cleaning up...
-------------------------------------------------------------------------------------------------------------
VMotion Interface <vmk0> was re-enabled
-------------------------------------------------------------------------------------------------------------

Offline

#4 2023-03-04 19:55:58

admin
Administrator
Registered: 2017-04-21
Posts: 2,057

Re: XSIBackup-DC regexp exclude vmdks help

Your REGEXP pattern is being rejected. Why are you using double parentheses?
Check this post: How to write regular expressions

Use something like this:

--exclude="[^/vmfs/volumes/datastorekubedata/kubevols*.]"

Offline

#5 2023-03-06 06:49:07

sdan
Member
Registered: 2023-03-02
Posts: 7

Re: XSIBackup-DC regexp exclude vmdks help

Hello

There is something wrong with your backup logic.

Please read the output of your script:

This vm testvm has 3 disks (each disk 20GB). 60 GB in total
scsi0:0.fileName = "testvm-000001.vmdk"
scsi0:1.fileName = "/vmfs/volumes/5d092ea7-c0c36652-bbd2-0017a477082c/kubestatic/testvm_1-000001.vmdk"
scsi0:2.fileName = "/vmfs/volumes/5d092ea7-c0c36652-bbd2-0017a477082c/kubevols/kubernetes-dynamic-pvc-testvm1212-000001.vmdk"

but your program wants to backup for this vm 4TB of data, the entire content of these 2 folders kubestatic and kubevols, not only their own vmdk,

We just bought 4 licenses of your product and we cannot use it.

|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.6.0.6: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
Shell binary pool memory size: 1912 MB
-------------------------------------------------------------------------------------------------------------
Shell memory pool size was set to 2048 MB by default
-------------------------------------------------------------------------------------------------------------
Total available memory: 249259 MB (~1GB is always kept as a security margin)
-------------------------------------------------------------------------------------------------------------
License: 000406F100000000000000000017a477083a | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
PID: 12002704, Running job as: root
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/datastoreringiernfs/esxivm
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
VMotion interface <vmk0> has been disabled
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/datastoreringiernfs/esxivm'
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: 37043876790272 (34.50 TB)
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
testvm Hardware Version is: 14
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: testvm
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM: testvm (powered on)...
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: testvm
-------------------------------------------------------------------------------------------------------------
Backup start date: 2023-03-06T06:36:10
-------------------------------------------------------------------------------------------------------------
The size of the data to backup is huge, more memory will be allocated
-------------------------------------------------------------------------------------------------------------
Memory size was automatically set to: 1912 MB
-------------------------------------------------------------------------------------------------------------
2023-03-06 06:36:10 | Backing up 135 files, total size is 4.90 TB

Offline

#6 2023-03-08 08:57:17

admin
Administrator
Registered: 2017-04-21
Posts: 2,057

Re: XSIBackup-DC regexp exclude vmdks help

As any other software (c)XSIBackup has to make some assumptions. In this case the asumption is that .vmdk disks are stored in folders, generally one, but could be more, and that the rest of files in those folders are other constituent files of the VM, namely: .vmsd, vmxf, snapshots, etc..., which are indeed backed up. Should any other file not referenced by the .vmx file be found there, it will also be backed up by default.

Now the thing is: why are there in these folders other .vmdk files which don't belong to the VM?. Maybe this is the source of the problem. Still you can exclude individual files based on patterns. Send us the contents of your VM folder clearly marking the files that you don't want to backup and we'll create the REGEP for you.

Offline

#7 2023-03-08 09:23:23

sdan
Member
Registered: 2023-03-02
Posts: 7

Re: XSIBackup-DC regexp exclude vmdks help

Hello

These 2 folders contains all the vmdk managed directly by the kubernetes cluster. These vmdks belongs to other vm nodes controlled by the kubernetes cluster. It does not conaint any vmx , log, .vmfs, vmxf files...
I will feel more safe when I will make the backup to completely exclude this type of folders...

I managed to find the right regexp I need it. but
I still get to each vm I want to backup
"2023-03-06 06:36:10 | Backing up 135 files, total size is 4.90 TB"
but the files are excluded...


vmfs/volumes/datastorelocal5/XSIBackup-DC/xsibackup \
--replica \
"VMs(testvm)" \
"/vmfs/volumes/datastoreringiernfs/esxivm" \
--disable-vmotion="" \
--use-smtp="1" \
--mail-to="sdan@globalis.ro" \
--exclude="(kubestatic|kubevols)"

Offline

#8 2023-03-08 13:30:28

admin
Administrator
Registered: 2017-04-21
Posts: 2,057

Re: XSIBackup-DC regexp exclude vmdks help

OK, great, but, did you finally achieved what you wanted?.
It seems as if you were just being misguided by the fact that all files are listed, then the excluded ones are marked as 'excluded', that's normal behavior.

Offline

#9 2023-03-09 08:02:50

sdan
Member
Registered: 2023-03-02
Posts: 7

Re: XSIBackup-DC regexp exclude vmdks help

Yes. I managed to do it, but...

In the next versions can you print this message after the exclusion is happening?

"2023-03-06 06:36:10 | Backing up xxx files, total size is XXX TB"

Now is misleading.

Offline

Board footer