Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

Thanks! No I would never dream of renaming any other way than through the Web UI and that takes care of renaming all other files as well, but anyway. There were two files that retained the old name, that one and an XML-file that just had <config /> in it. Shut down the VM. Renamed those two and restarted. Backup is now running......

Thanks for that explanation, makes sense. Anyway, our ESX hosts are not exposed to the outside world. Only hosts that need to, ie webservers and stuff and never port 22

OK. Noted. FWIW has your software worked flawlessly since September last year, but we will evaluate an upgrade.

"(c)XSIBackup-Pro suppports spaces in file names."

Any yet it would, on every second go complain that the file was not there:

2020-12-25T18:14:03.596Z info hostd[2098563] [Originator@6876 sub=DiskLib opID=esxui-8168-138b user=root] DISKLIB-DSCPTR: DescriptorOpenInt: failed to open '/vmfs/volumes/5f97f8f4-7790c2f6-ea6b-74867aed9da0/Filr4/Novell Filr-2_1.vmdk': Could not find the file (60002)

Instead of the two files, there would be just one, named "Novell.vdmk"

I could clearly see that it cut off the name after the space. 100% repeatable. Could it have been a limitation of RSync?

Anyway. It works after the rename.

OK. Here is the tl;dr summary:

One of the disk files in that VM had a space in the filename.

The long story:
The VM is an appliance called "Filr". It is a file sync/publishing appliance. A little bit like Sharepoint/OneDrive, but with the added benefit that files are in-house so you can be GDPR-compliant.

We have been using this for a long time and the VM probably started out running on VMWare 4. It has been upgraded several times, but from version 2 and forward you upgraded by installing a new appliance and added the data disk from the old. Thus, the data disk kept the name it was asutomatically given when the Filr2 installation was created "Novell Filr-2_1.vmdk".

This worked fine for XSIBackup when running from disk A to B on the same host, but not from one host to another. The transfer ended up first transferring "Novell Filr-2_1.vmdk" as "Novell.vmdk" then came "Novell Filr-2_1-flat.vmdk" and overwrote as "Novell.vmdk".

My initial mistake when troubleshooting was that I looked at the hostd.log on the source, when I should have looked at the target. I misunderstood the wording of the error message: "ERROR DIFDELAL, details: [Filr4] error: error deleting all snapshots VM [Filr4_XSIBAK] Id 29, error: Remove All Snapshots". When I read it now I definitely see that it uniquely points to the target, but then again is hindsight always 20-20.

Hope this helps someone else!

I strongly suspect that the issue is with one of the "disks" of that VM.

So, if I look at hostd.log on the target, I definitely see things:

2020-12-25T09:15:33.297Z info hostd[2099162] [Originator@6876 sub=Libs opID=vim-cmd-2d-b471 user=root] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/5f97f8f4-7790c2f6-ea6b-74867aed9da0/Filr4/Novell Filr-2_1.vmdk' : Could not find the file (393218).
2020-12-25T09:15:33.297Z info hostd[2099162] [Originator@6876 sub=DiskLib opID=vim-cmd-2d-b471 user=root] DISKLIB-DSCPTR: DescriptorOpenInt: failed to open '/vmfs/volumes/5f97f8f4-7790c2f6-ea6b-74867aed9da0/Filr4/Novell Filr-2_1.vmdk': Could not find the file (60002)
2020-12-25T09:15:33.298Z info hostd[2099162] [Originator@6876 sub=DiskLib opID=vim-cmd-2d-b471 user=root] DISKLIB-LINK  : "/vmfs/volumes/5f97f8f4-7790c2f6-ea6b-74867aed9da0/Filr4/Novell Filr-2_1.vmdk" : failed to open (The system cannot find the file specified).
2020-12-25T09:15:33.298Z info hostd[2099162] [Originator@6876 sub=DiskLib opID=vim-cmd-2d-b471 user=root] DISKLIB-CHAIN :"/vmfs/volumes/5f97f8f4-7790c2f6-ea6b-74867aed9da0/Filr4/Novell Filr-2_1-000002.vmdk": Failed to open parent "/vmfs/volumes/5f97f8f4-7790c2f6-ea6b-74867aed9da0/Filr4/Novell Filr-2_1.vmdk": The system cannot find the file specified.
2020-12-25T09:15:33.298Z info hostd[2099162] [Originator@6876 sub=DiskLib opID=vim-cmd-2d-b471 user=root] DISKLIB-CHAIN : "/vmfs/volumes/5f97f8f4-7790c2f6-ea6b-74867aed9da0/Filr4/Novell Filr-2_1.vmdk" : failed to open (The parent of this virtual disk could not be opened).
2020-12-25T09:15:33.298Z info hostd[2099162] [Originator@6876 sub=DiskLib opID=vim-cmd-2d-b471 user=root] DISKLIB-LIB   : Failed to open '/vmfs/volumes/5f97f8f4-7790c2f6-ea6b-74867aed9da0/Filr4/Novell Filr-2_1-000002.vmdk' with flags 0x3 The parent of this virtual disk could not be opened (23).

So, why this Filr4/Novell Filr-2_1-000002.vmdk? That file exists on the target but not on source. Also present on the target is several Filr4-Snapshotnnnn files.

I also see the remove snapshot erroro on the target server in the UI. I will move the datastore folder on the target to another disk and retry the backup operation

Tried first with the service shut down (in between preparing Christmas dinner). Fails on every second backup. Then turned off the VM completely. Still fails, so the error must be somewhere else?

The server in Question is for file sync. It is only accessed by clients. I will start by breaking out the backup to a separate job at another time which will make comparing logs easier. I will get back to you if I cannot solve this on our own.

Well I see the SoapAdapter.HTTPService error both for failed backups and backups that run OK. Also is the fact that it happens exactly every other day interesting. There is no activity that I know of at this time, but I could change the time and see if it makes any difference.

Would it help if I mailed you the log?

I could really use some help here. It is absolutely repeatable. It fails every other day. Is there ano other logs I could look at? In hostd I see an entry "Task Created : haTask-24-vim.VirtualMachine.removeAllSnapshots" on the days when it works, but not on the days it does not. On days when it fails the first access for that VM seems to be "Task Created : haTask-24-vim.ManagedEntity.reload"

Well, I posted the relevant snippet from the log earlier Anyway. The error might be related to something stopping ESXi from quiescing the machine. I will investigate further.

And Yes now that we have narrowed it down, I can troubleshoot further.