#1 2018-05-23 13:49:06

clicream
Member
Registered: 2018-04-19
Posts: 27

Independent disks cannot be hot backed up

xsibackup pro 10.3.4

I'm getting vmdk lock and independent disk error but it is dependent. How to fix this?

2018-05-23T10:10:38|  The file "/vmfs/volumes/Datastore-RAID10/MAIL2/MAIL2_3-000002-delta.vmdk" is locked. Independent disks cannot be hot backed up, issue a cold backup for or exlude the disk from the backup. Make sure that the VM state (on/off) was not manually changed during the backup
ERROR: cannot locate .ERR file at [/vmfs/volumes/C220-mailesxi02/xsibackup-dir/.ERR-fe619c1144925223ca7eb164fd5870ff56ae7eb4]

Offline

#2 2018-05-23 14:51:27

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: Independent disks cannot be hot backed up

The file cannot be read, that's why you are getting this error.
It cannot be read because some other process is holding it.
The way to find which other process is holding the file is to run this command right before you get that error:

ps -c | grep "$( lsof | grep "MAIL2/MAIL2_3-000002-delta.vmdk" | awk '{print $1}' )"

It will most probably be open by the VM process, so make sure VMWare Tools are installed and running and that you meet all the requirements for the system to be quiesced properly. If you are running a Mail server, it will most probably be using a database too. If it's a Windows server, make sure you have all main services configured the right way:

https://33hops.com/troubleshooting-wind … -esxi.html

The database server might need additional dependent services to be able to quiesce it along with the OS FS.
We have found Windows servers under heavy load (in the middle of a file level backup) to still be problematic to quiesce.

In case you cannot get your OS to be quiesced properly and you can asume some seconds of downtime, you can use warm backup type, which will ensure all files are backed up in a consistent state.

Offline

#3 2018-05-23 15:12:49

clicream
Member
Registered: 2018-04-19
Posts: 27

Re: Independent disks cannot be hot backed up

these are output that I got

ps -c | grep "$( lsof | grep "MAIL2/MAIL2_3-000002-delta.vmdk" | awk '{print
$1}' )"
20477343  20477343  vmx                            /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477353  20477343  vmx-vthread-12                 /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477354  20477343  vmx-vthread-13:MAIL2           /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477355  20477343  vmx-vthread-14:MAIL2           /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477356  20477343  vmx-vthread-15:MAIL2           /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477357  20477343  vmx-mks:MAIL2                  /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477358  20477343  vmx-svga:MAIL2                 /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477359  20477343  vmx-vcpu-0:MAIL2               /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477360  20477343  vmx-vcpu-1:MAIL2               /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477361  20477343  vmx-vcpu-2:MAIL2               /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477362  20477343  vmx-vcpu-3:MAIL2               /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477363  20477343  vmx-vcpu-4:MAIL2               /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477364  20477343  vmx-vcpu-5:MAIL2               /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477365  20477343  vmx-vcpu-6:MAIL2               /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx
20477366  20477343  vmx-vcpu-7:MAIL2               /bin/vmx -s sched.group=host/user -# product=2;name=VMware ESX;version=6.0.0;buildnumber=4192238;licensename=VMware ESX Server;licenseversion=6.0; -@ transport=3;msgs=ui /vmfs/volumes/59f9401a-c459a0c2-6d8a-70708b1ed678/MAIL2/MAIL2.vmx

VM is a Zimbra mail server that is using mariadb database
It have vmware-tools installed.

Offline

#4 2018-05-23 16:36:57

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: Independent disks cannot be hot backed up

Please, post the backup command you are using so that we know something more.

Well, as expected its the /bin/vmx process that is holding the file.
The most probable cause is some VM malfunction that 99% of the times is helped by:

1/ Deleting all snapshots and rebooting the VM.
2/ Restarting the host service: # /etc/init.d/hostd restart
3/ Rebooting the host.

You should take action in the above order until you fix the issue.

If you want to try to dig a bit more into the problem:

1 - Delete all snapshots.
2 - Take a quiesced snapshot (no memory included). If you can't, then there's some issue with the VMWare Tools service, follow advice above.
3 - Inspect the VM configuration while the VM is running and see what are the disk names the VM is running on, they should be:

MAIL2-000001.vmdk
MAIL2_1-000001.vmdk
MAIL2_2-000001.vmdk
MAIL2_3-000001.vmdk
...

We are again guessing, but as the problematic snapshot is named 000002, is it correct to suppouse you are using OneDiff to backup your VM?

If not, you should wonder why you have two snapshots present instead of just the backup snapshot.

Offline

#5 2018-05-23 16:49:20

clicream
Member
Registered: 2018-04-19
Posts: 27

Re: Independent disks cannot be hot backed up

I had problem with snapshot before where I cancelled backup delete halfway and ended up have to edit vmx file to skip a snapshot to be able to boot up that VM (MAIL2)

Offline

#6 2018-05-23 18:44:55

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: Independent disks cannot be hot backed up

Then do make sure that you have recomposed your VMX file the way it should be. Being able to run a VM under ESXi does not neccessarily mean that it's correctly configured. You might be running one of the disks on top of a snapshot.

Offline

#7 2018-05-24 06:02:42

clicream
Member
Registered: 2018-04-19
Posts: 27

Re: Independent disks cannot be hot backed up

So I need to reconcile vm snapshot first?
Because when a backup schedule failed, I found that hard disk is pointed to -0001.vmdk file instead of just MAIL2_1.vmdk

Offline

#8 2018-05-24 09:16:14

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: Independent disks cannot be hot backed up

We don't even know if you are able to take and delete a snapshot so far.
That depends on what kind of backup you want to perform, but as you have doubts about the state of your VM, first thing to do is delete all and make sure you can create and delete snapshots before going any further.
You cannot drive a car if you are not sure whether it has all wheels on or there some missing.

Offline

#9 2018-05-24 09:23:10

clicream
Member
Registered: 2018-04-19
Posts: 27

Re: Independent disks cannot be hot backed up

I'm able to reconcile snapshots but did you mean delete from within VM folder?

Offline

#10 2018-05-24 14:35:32

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: Independent disks cannot be hot backed up

Deleting a snapshot does not mean that the data contained in it will be deleted, deleting a snapshot merges data in it into the base disks, but this is common terminology.

So, delete the snapshot, merge the data in the snapshot or reconcile it, and once you have performed that operation, do make sure you can create quiesced snapshots and delete them before even trying to make a backup.

Should your VM become rebelious (that could happen if VMWare Tools cannot handle MariaDB properly) due to heavy load during the snapshot operation, ponder whether performing a non-quiesced snapshot or a warm one. You have many other possibilities before going for non-quiesced or warm in any case. Just delve into how MariaDB handles data and how it interacts with LVM and VMWare Tools during the quiescing process.

Offline

Board footer