#1 2019-01-06 01:19:21

DaveK
Member
Registered: 2018-02-10
Posts: 15

ERROR DIFDELAL on snapshot remove

After a few months of fully successful backup on ESXi 6.5 (Build 7967591) I, unfortunately, got errors while making OneDiff copies. If I remove backups from the destination and make new initial backup everything works well, but the next backup fails while XSI tries to remove a snapshot from newly created _XSIBack, no matter from the source VM is powered on, or off. Obviously, I tried with host reboot, checking free space etc. also I have checked these solutions: https://33hops.com/esxi-snapshot-errors … tions.html. Source VM before make initial backup has no snapshots, and has only one vmdk. After spending a lot of time on various testing I've got no clue, what happened. Please for help.

Log logfile and mail below:

2019-01-06T00:40:23|  VMs to backup:
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:23|  44     Ubuntu        [datastore1] Ubuntu/Ubuntu.vmx                ubuntu64Guest           vmx-08                   
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:25|  Calculating sizes...
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:26|  Needed room: 26 Gb.
2019-01-06T00:40:26|  Sparse size: 100 Gb.
2019-01-06T00:40:26|  Available room: 1207 Gb.
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:27|  [Ubuntu] Starting backup (size is 27000M on 102400M file)
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:27|  XSIBackup will backup your VMs while they are running and will quiesce guest services too, so that users
2019-01-06T00:40:27|  can continue to use the VM while the backup is taking place. You can also run cold and warm --backup-how
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:27|  Hot backup selected for VM: [Ubuntu], the VM is off though
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:27|  [Ubuntu] info: boot partition is MBR
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:28|  [Ubuntu] info: quick size check...
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:28|  [Ubuntu] info: file [Ubuntu-flat.vmdk]...
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:28|  [Ubuntu] info: file size check | OK [ 107374182400 bytes | 107374182400 bytes ]
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:28|  [Ubuntu] info: remote VMX file size 3232
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:29|  System disk CIDs 60b707d2|60b707d2
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:29|  [Ubuntu] (c) OneDiff algorithm
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:29|  Mirror VM exists and the system disks CID are the same, starting OneDiff...
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:29|  [Ubuntu] info: length of .vmx file before OneDiff phase is 3231
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:29|  Snapshot & Quiescing
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:29|  [Ubuntu] info: the VM will not be quiesced, no tools detected
2019-01-06T00:40:29|  [Ubuntu] info: round 2
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:35|  [Ubuntu] info: snapshot taken, quiescing status: NO (no tools detected)
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:36|  Initiating OneDiff differential backup...
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:36|  [Ubuntu] info: delta remains cleaned up at mirror VM [Ubuntu_XSIBAK]
---------------------------------------------------------------------------------------------------------------------------------
[Ubuntu] info: transfering snapshot file [Ubuntu-Snapshot1.vmsn]
[32m(c) XSIDiff 1.1.1.4 licensed:
[0mA full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/datastore1/Ubuntu/Ubuntu-Snapshot1.vmsn
Writing to file /vmfs/volumes/datastore_d2/VM_BACKUP/Ubuntu/Ubuntu-Snapshot1.vmsn
Set input file offset to 0 bytes

100.00% done
Time taken: 1 seconds
Speed 1.00 mb/s
Overall speed 0.02 mb/s
---------------------------------------------------------------------------------------------------------------------------------
[Ubuntu] info: transfering snapshot file [Ubuntu-000001.vmdk]
[32m(c) XSIDiff 1.1.1.4 licensed:
[0mA full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/datastore1/Ubuntu/Ubuntu-000001.vmdk
Writing to file /vmfs/volumes/datastore_d2/VM_BACKUP/Ubuntu/Ubuntu-000001.vmdk
Set input file offset to 0 bytes

100.00% done
Time taken: 1 seconds
Speed 1.00 mb/s
Overall speed 0.00 mb/s
---------------------------------------------------------------------------------------------------------------------------------
[Ubuntu] info: transfering snapshot file [Ubuntu-000001-delta.vmdk]
[32m(c) XSIDiff 1.1.1.4 licensed:
[0mA full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/datastore1/Ubuntu/Ubuntu-000001-delta.vmdk
Writing to file /vmfs/volumes/datastore_d2/VM_BACKUP/Ubuntu/Ubuntu-000001-delta.vmdk
Set input file offset to 0 bytes

100.00% done
Time taken: 1 seconds
Speed 1.00 mb/s
Overall speed 0.20 mb/s
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:37|  [Ubuntu] info: length of remote .vmx file after copying it is 3231
2019-01-06T00:40:38|  [Ubuntu] info: length of remote .vmx after replacing name is 3238
2019-01-06T00:40:38|  [Ubuntu] info: name changed to [Ubuntu_XSIBAK]
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:38|  Removing hot snapshot & merging data
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:39|  [Ubuntu] info: OneDiff backup differential data (1) merged with base data
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:40|  [Ubuntu] info: the backup mirror VM [Ubuntu_XSIBAK] Id 45 has been created
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:40|  [Ubuntu] info: diff mirror VM reloaded
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:40|  [Ubuntu] info: deleting snapshot at [Ubuntu_XSIBAK] Id 45...
2019-01-06T00:40:40|  [Ubuntu] error DIFDELAL: error deleting all snapshots VM [Ubuntu_XSIBAK] Id 45, details: Remove All Snapshots:
2019-01-06T00:40:40|  Remove all snapshots failed
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:40|  [Ubuntu] info: Updating CID at [/vmfs/volumes/datastore_d2/VM_BACKUP/Ubuntu/Ubuntu.vmdk]
2019-01-06T00:40:40|  [Ubuntu] info: finished OneDiff backup, CID updated
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:40|  [Ubuntu] info: performing quick check...
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:40|  [Ubuntu] info: file [Ubuntu-flat.vmdk]...
2019-01-06T00:40:40|  [Ubuntu] info: file size check | OK [ 107374182400B | 107374182400B ]
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:40|  [Ubuntu] info: comparing first 500M of [Ubuntu-flat.vmdk]...
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:46|  [Ubuntu] info: (source) first 500M hash is: e4d996124d9b971bc42cc0e170c70c97a5139b10
2019-01-06T00:40:46|  [Ubuntu] info: (target) first 500M hash is: e4d996124d9b971bc42cc0e170c70c97a5139b10
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:46|  [Ubuntu] info: first 500M check | OK [ e4d996124d9b971bc42cc0e170c70c97a5139b10 ]
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:46|  [Ubuntu] info: used size check | OK [ 26998M ]
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:59|  The ESXi configuration was saved to "/vmfs/volumes/datastore_d2/VM_BACKUP"
---------------------------------------------------------------------------------------------------------------------------------
2019-01-06T00:40:59|  Errors detected in backup, check logs
---------------------------------------------------------------------------------------------------------------------------------


From mail:

Done hot backup (id: ub_local) using onediff (no compression)
Available room in device /vmfs/volumes/datastore_d2/VM_BACKUP before backup: 1207 Gb.
Sparse size on disk of the selected virtual machines: 100 Gb.
Needed room in device /vmfs/volumes/datastore_d2/VM_BACKUP for backup: 26 Gb.
(Id)VM Name    State    Size (Gb)    Stop    Copy    Start    Time (min)    Speed (mb/s)
(44) Ubuntu    OFF    26/ 100    Off yet    KO!    -    0    870/ 3303
Last error raised for the above VM:
ERROR DIFDELAL, details: [Ubuntu] error: error deleting all snapshots VM [Ubuntu_XSIBAK] Id 45, details: Remove All Snapshots: Remove all snapshots failed
Available space in device /vmfs/volumes/datastore_d2/VM_BACKUP after backup: 1207 Gb.
Complete backup elapsed time: 0 min
The ESXi configuration was saved to /vmfs/volumes/datastore_d2/VM_BACKUP
• [ Sun Jan 6 00:40:40 UTC 2019 ] ERROR (DIFDELAL), details [Ubuntu] error: error deleting all snapshots VM [Ubuntu_XSIBAK] Id 45, details: Remove All Snapshots:
• Remove all snapshots failed

Offline

#2 2019-01-06 13:21:47

admin
Administrator
Registered: 2017-04-21
Posts: 737

Re: ERROR DIFDELAL on snapshot remove

ERROR DIFDELAL, details: [Ubuntu] error: error deleting all snapshots VM [Ubuntu_XSIBAK] Id 45, details: Remove All Snapshots: Remove all snapshots failed

This is an ESXi error, it's not XSIBackup related. If you try to delete the snapshots from the GUI you may find out you can't, or you may be able, that message just states that ESXi was not able to delete them at the time to perform the backup.

ESXi is great but it's far from perfection. If you did some works in the host, like adding some new VMs or reconfiguring something in the guest, we have noticed the snapshot subsystem may get unstable. Just reset the OneDiff cycle.

Offline

#3 2019-01-06 14:04:50

DaveK
Member
Registered: 2018-02-10
Posts: 15

Re: ERROR DIFDELAL on snapshot remove

When I try to remove all snapshots from Ubuntu _XSIBackup VM manually from ESXi web interface I've got an error:

Failed - A general system error occurred: vim.fault.GenericVmConfigFault


When I delete Ubuntu_XSIBackup VM from GUI he left in VM directory two files, looks like they are not treated as part of VM:
Ubuntu-000002-delta.vmdk
Ubuntu-Snapshot1.vmsn

In ESXi Web GUI Ubuntu_XSIBackup has disk [datastore_d2] VM_BACKUP/Ubuntu/Ubuntu-000002.vmdk, capacity 0 B

Also, I noticed Ubuntu_XSIBackup has no VMX file, only "Ubuntu.vmx.bak" after renaming to Ubuntu.vmx i can reregister VM, but can't start, error:

Cannot open the disk '/vmfs/volumes/5b5fff52-bf239984-debc-a4bf012da28c/VM_BACKUP/Ubuntu/Ubuntu-000001.vmdk' or one of the snapshot disks it depends on.

So it looks like a problem with the proper chain of snapshots, but this happens only with snapshots created with XSI OneDiff copied VM's. Snapshots of source VM's are created and deleted without any problems, also manual snapshots. What was could suddenly happen to ESXi snapshot subsystem, that hits only XSI functions, what can I do more?

Ubuntu.vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=40abdb0a
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"

# Extent description
RW 209715200 VMFS "Ubuntu-flat.vmdk"

# The Disk Data Base
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "13054"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "0a2e13619a158f2bc2a6664e40abdb0a"
ddb.thinProvisioned = "1"
ddb.uuid = "60 00 C2 9b 59 e9 d4 56-52 24 f9 3d d0 88 8b 71"
ddb.virtualHWVersion = "8"




Ubuntu-000001.vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=40abdb0a
parentCID=8a1e6dd6
isNativeSnapshot="no"
createType="vmfsSparse"
parentFileNameHint="Ubuntu.vmdk"
# Extent description
RW 209715200 VMFSSPARSE "Ubuntu-000001-delta.vmdk"

# The Disk Data Base
#DDB

ddb.longContentID = "0a2e13619a158f2bc2a6664e40abdb0a"






Ubuntu.vmsd
.encoding = "UTF-8"
snapshot.lastUID = "1"
snapshot.current = "1"
snapshot0.uid = "1"
snapshot0.filename = "Ubuntu-Snapshot1.vmsn"
snapshot0.displayName = "xsibackupdiff"
snapshot0.description = "xsibackupdiff 1"
snapshot0.createTimeHigh = "360137"
snapshot0.createTimeLow = "-1675272195"
snapshot0.numDisks = "1"
snapshot0.disk0.fileName = "Ubuntu.vmdk"
snapshot0.disk0.node = "scsi0:1"
snapshot.numSnapshots = "1"

Last edited by DaveK (2019-01-06 14:11:55)

Offline

#4 2019-01-06 15:53:57

admin
Administrator
Registered: 2017-04-21
Posts: 737

Re: ERROR DIFDELAL on snapshot remove

- .vmx file is renamed to .vmx.bak to reset the (c)OneDiff cycle automatically.
- As the .vmx file is not present, rename .vmx.bak back to .vmx (you already did).
- Error: error deleting all snapshots VM. This is just a generic error, you must follow the thread to know what's the root cause. Inspect your ESXi log files to delve into the issue.
- All needed files seem to be present: -000001.vmdk, -000001-delta.vmdk, .vmsn
- It could be that some process is holding some of the snapshot files. Use lsof command to find out:

lsof | grep "datastore_d2/VM_BACKUP/Ubuntu"

- Reboot the host if you have the chance, this does magic most of the times.

Offline

#5 2019-01-06 19:45:11

DaveK
Member
Registered: 2018-02-10
Posts: 15

Re: ERROR DIFDELAL on snapshot remove

Maybe that is a generic error, but why only on copies that were done by XSI? No process doesn't lock snapshot files. When I wipe Ubuntu.vmsd content and made a change in VMX file (change disk file name to Ubuntu.vmdk), creating and deleting new snapshots works well. So I assume "snapshot subsystem" must work well. The only snapshots created while OneDiff copy doesn't work, is it do that in another way than a manual snapshot from GUI?
BTW esxi host was rebooted many times.

Last edited by DaveK (2019-01-06 19:58:03)

Offline

#6 2019-01-06 20:49:32

admin
Administrator
Registered: 2017-04-21
Posts: 737

Re: ERROR DIFDELAL on snapshot remove

By wiping the .vmsd file and pointing to the base disk you are discarding the data in the snapshot. What you describe is normal behaviour, as the base disk was copied without errors, thus it will work if you use it as the boot disk, nothing out of usual.

You should delve into the stack of errors by searching in the ESXi host logs and find the ultimate cause of the snapshot coalescing problem. I doubt that the -delta.vmdk file is corrupt, it is in fact the only file neccessary to integrate the snapshot data, but that's only from a pure data analisys point of view, you do need the auxiliary files, at least .vmsd to be able to delete it.

Offline

#7 2019-01-07 00:16:01

DaveK
Member
Registered: 2018-02-10
Posts: 15

Re: ERROR DIFDELAL on snapshot remove

And finally, I found "The bug" Problem was in the file system, source datastore is VMFS5 destination VMFS6. So XSI made snapshots in another way, using a mechanism that not supported, or incompatible with VMFS6. After moving VM to VMFS6 everything works well. I don't find any mention about that in the documentation.

Thanks for a good point to look at the logs. In hostd.log I found that lines:

DiskLib_IsVMFSSparseSupported: vmfssparse is not supported on /vmfs/volumes/5b5fff52-bf239984-debc-a4bf012da28c/VM_BACKUP/Ubuntu: f532.
DISKLIB-VMFS_SPARSE : Failed to open vmfssparse disk /vmfs/volumes/5b5fff52-bf239984-debc-a4bf012da28c/VM_BACKUP/Ubuntu/"Ubuntu-000001-delta.vmdk": The specified feature is not supported by this version (24).
DISKLIB-LINK  : "/vmfs/volumes/5b5fff52-bf239984-debc-a4bf012da28c/VM_BACKUP/Ubuntu/Ubuntu-000001.vmdk" : failed to open (The specified feature is not supported by this version).
DISKLIB-CHAIN : "/vmfs/volumes/5b5fff52-bf239984-debc-a4bf012da28c/VM_BACKUP/Ubuntu/Ubuntu-000001.vmdk" : failed to open (The specified feature is not supported by this version).
DISKLIB-LIB   : Failed to open '/vmfs/volumes/5b5fff52-bf239984-debc-a4bf012da28c/VM_BACKUP/Ubuntu/Ubuntu-000001.vmdk' with flags 0x13 The specified feature is not supported by this version (24).
Load failed: cannot open /vmfs/volumes/5b5fff52-bf239984-debc-a4bf012da28c/VM_BACKUP/Ubuntu/Ubuntu-000001.vmdk (24): The specified feature is not supported by this version

Offline

#8 2019-01-07 11:02:17

admin
Administrator
Registered: 2017-04-21
Posts: 737

Re: ERROR DIFDELAL on snapshot remove

You have a big notice with a big alert sign at the top of the feature page, which has a direct link from the product page:

https://33hops.com/xsibackup-pro-onediff.html

On the other side you cut down the file system information from the log, so we coudn't help you any further.

It's not necessary to explain why OneDiff can't be used accross incompatible file systems, it's a kind of obvious that exchanging snapshot data between VMs hosted in different FS won't work. In any case, we never loose the chance to remind that in every post covering OneDiff.

Offline

#9 2019-01-07 11:40:20

DaveK
Member
Registered: 2018-02-10
Posts: 15

Re: ERROR DIFDELAL on snapshot remove

In fact, there is information about it, my fault. But probably warning or error in a log while XSI starts and checks source and destination datastore and VMs will be more helpful and time-saving of frustrated users;)
Thanks for help and involvement.

Offline

#10 2019-01-07 12:21:16

admin
Administrator
Registered: 2017-04-21
Posts: 737

Re: ERROR DIFDELAL on snapshot remove

No worries. There are indeed warnings around mixed HW versions. We'll revise the checks around file systems to refine it as much as possible.

Offline

Board footer