#1 2020-06-28 13:09:30

sphen
Member
Registered: 2020-06-13
Posts: 45

Fails to create snapshot even though it reports success

How would you suggest I troubleshoot this issue.  I have successfully set up xsibackup DC on 3 hosts and backups are going well. On a single VM, xsibackup fails to create a snapshot and thus does not back up the vmdk files as they are excluded/in-use. The verbose output does not report an error and if i manually create a snapshot prior to backup, the backup runs because a previous snapshot exists.

output below is truncated

[root@localhost:/vmfs/volumes/5d75000d-393caa77-6cf5-a0369f3ea540/XSI] /vmfs/volumes/QNAP3/XSI/pro1/XSIBackup-DC/xsibackup --backup "VMs(Global Viewer I)" admin@172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/ --ssh-ciphers=aes128-gcm@openssh.com --remote-path=/share/CACHEDEV1_DATA/BKUP/xsibackup --verbosity=10
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-----------------------------------------------------------------------------------------------------------
License: XXXX
-----------------------------------------------------------------------------------------------------------
Primary TMP folder: /tmp/xsi/2307404
-----------------------------------------------------------------------------------------------------------
Secondary TMP folder: /vmfs/volumes/QNAP3/XSI/pro1/XSIBackup-DC/tmp/2307404
-----------------------------------------------------------------------------------------------------------
SSH: Sun Jun 28 08:57:11 EDT 2020
-----------------------------------------------------------------------------------------------------------
Remote system: linux
-----------------------------------------------------------------------------------------------------------
User defined ciphers: aes128-gcm@openssh.com
-----------------------------------------------------------------------------------------------------------
PID: 2307404, Running job as: admin
-----------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-----------------------------------------------------------------------------------------------------------
Block size is 10485760 bytes (10.00 MB)
-----------------------------------------------------------------------------------------------------------
Configured ciphers: aes128-gcm@openssh.com
-----------------------------------------------------------------------------------------------------------
Checking whether XSIBackup-DC can be found at </share/CACHEDEV1_DATA/BKUP/xsibackup>
-----------------------------------------------------------------------------------------------------------
XSIBackup-DC was found at </share/CACHEDEV1_DATA/BKUP/xsibackup>
-----------------------------------------------------------------------------------------------------------
Created remote folder at 172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/20200628125711
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC setting repository at /share/CACHEDEV1_DATA/BKUP/
-----------------------------------------------------------------------------------------------------------
Found remote dir /share/CACHEDEV1_DATA/BKUP//data at 172.20.30.40:22
-----------------------------------------------------------------------------------------------------------
.xsitools file located at admin@172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/.xsitools
Ordering blocks 100.00%
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: Global Viewer I
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : Global Viewer I (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
Created remote folder at 172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/20200628125711/Global Viewer I
-----------------------------------------------------------------------------------------------------------
Lock file: /share/CACHEDEV1_DATA/BKUP/20200628125711/Global Viewer I/.locked
New backup/replica: Global Viewer I | folder
-----------------------------------------------------------------------------------------------------------
Backup start date: 2020-06-28 12:57:23
-----------------------------------------------------------------------------------------------------------
2020-06-28 12:57:23 | Backing up 22 files, total size is 545567757997
-----------------------------------------------------------------------------------------------------------
    NUMBER	                                                   FILE             SIZE       PROGRESS
-----------------------------------------------------------------------------------------------------------
     1/22	                          Global Viewer I-0459a943.hlog    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
     2/22	                              Global Viewer I-flat.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
     3/22	                                   Global Viewer I.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
     4/22	                                  Global Viewer I.nvram    [skipped excluded]
....

with manual snapshot:

[root@localhost:/vmfs/volumes/5d75000d-393caa77-6cf5-a0369f3ea540/XSI] /vmfs/volumes/QNAP3/XSI/pro1/XSIBackup-DC/xsibackup --backup "VMs(Global Viewer I)" admin@172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/ --ssh-ciphers=aes128-gcm@openssh.com --remote-path=/share/CACHEDEV1_DATA/BKUP/xsibackup
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-----------------------------------------------------------------------------------------------------------
License: XXXX
-----------------------------------------------------------------------------------------------------------
SSH: Sun Jun 28 08:56:00 EDT 2020
-----------------------------------------------------------------------------------------------------------
Remote system: linux
-----------------------------------------------------------------------------------------------------------
User defined ciphers: aes128-gcm@openssh.com
-----------------------------------------------------------------------------------------------------------
PID: 2307266, Running job as: admin
-----------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-----------------------------------------------------------------------------------------------------------
Block size is 10485760 bytes (10.00 MB)
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC setting repository at /share/CACHEDEV1_DATA/BKUP/
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: Global Viewer I
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : Global Viewer I (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
New backup/replica: Global Viewer I | folder
-----------------------------------------------------------------------------------------------------------
Backup start date: 2020-06-28 12:56:12
-----------------------------------------------------------------------------------------------------------
2020-06-28 12:56:12 | Backing up 26 files, total size is 556295208637
-----------------------------------------------------------------------------------------------------------
    NUMBER	                                                   FILE             SIZE       PROGRESS
-----------------------------------------------------------------------------------------------------------
     1/26                                 Global Viewer I-0459a943.hlog         92.00 B  | Done   0.00%
-----------------------------------------------------------------------------------------------------------
     2/26                                     Global Viewer I-flat.vmdk        500.00 GB | Done   0.00%
...

Offline

#2 2020-06-28 14:23:10

admin
Administrator
Registered: 2017-04-21
Posts: 1,364

Re: Fails to create snapshot even though it reports success

There isn't any sign that the snapshot is failing to be taken. Can you see something denoting that in the ESXi log?
Nonetheless the -flat.vmdk file does indeed seem to be blocked.

Check whether there is some .lck file on the flat file from some previous interrupted process and delete it manually should it exist.

Offline

#3 2020-06-30 11:16:09

sphen
Member
Registered: 2020-06-13
Posts: 45

Re: Fails to create snapshot even though it reports success

So far I have not found anything to explain it - still looking. There are no .lck files and vmkfstools -D doesnt show a lock on the vmdk file. Either way I can create and remove snapshots manually on the VM. I do not see any errors in logs.

Offline

#4 2020-07-01 10:30:03

admin
Administrator
Registered: 2017-04-21
Posts: 1,364

Re: Fails to create snapshot even though it reports success

Is that file in a cluster?, is it being open by some other process?
What kind of DS is it residing in?
Something is clearly holding it at the time to backup.
What type of guest OS are you running?.

Offline

#5 2020-07-01 10:33:24

sphen
Member
Registered: 2020-06-13
Posts: 45

Re: Fails to create snapshot even though it reports success

yeah it is odd - i figure i will go through shutting it down - unregistering and re-registering when i get a moment.

it is in a cluster but all other vms are as well. on VMFS6 and Win 10. I just wish i could see an actual error

Offline

#6 2020-07-01 10:43:36

sphen
Member
Registered: 2020-06-13
Posts: 45

Re: Fails to create snapshot even though it reports success

well whatever it was has been cleared up by unregister/re-register. thx.

Offline

#7 2020-07-01 10:47:41

admin
Administrator
Registered: 2017-04-21
Posts: 1,364

Re: Fails to create snapshot even though it reports success

There's been errors related to snapshots in Windows since ever. Now in more recent versions of ESXi and Windows the problems not only didn't cease, but they seem to have managed to create a bunch of new ones.

https://kb.vmware.com/s/article/71130
https://kb.vmware.com/s/article/60395

Whatever is holding the file, it is outside the guest.
What type of Datastore are you using?
It might not be a cluster lock that you are experiencing.

Offline

#8 2020-07-01 11:00:15

sphen
Member
Registered: 2020-06-13
Posts: 45

Re: Fails to create snapshot even though it reports success

i see. the odd thing is that i was able to make a snapshot via vcenter when xsibackup failed to. it is on iSCSI vmfs6. it seems to be working now once it re-registered.

Offline

#9 2020-07-01 11:13:26

admin
Administrator
Registered: 2017-04-21
Posts: 1,364

Re: Fails to create snapshot even though it reports success

That doesn't mean anything, the snapshot could be failing due to some heavy load during the backup per instance, or some missing software in the guest, like VMWare Tools, or even some faulty service that causes the snapshot request to time out.

You should check the guest's logs more than the ESXi host itself when you have this kind of issue.

Offline

Board footer