#1 2019-12-17 10:42:02

lievenmoors
Member
Registered: 2018-11-21
Posts: 28

backup of VM failed with "ERROR: cannot locate .ERR file"

I get an error after which the backup of one in three VM's gets aborted with:

ERROR: cannot locate .ERR file at [/vmfs/volumes/datastore1/xsi-dir/.ERR-83d64b47e3d6103d7e158cc8b48a1c43c7097e1d]
2019-12-16T23:01:00|  [terra] CLXSITO1 error: Killed
2019-12-16T23:01:00|  sh: bad number

Below, you will find the xsibackup log output until that error.

Do you have an idea of what could have gone wrong here?

Thanks for your help,

lieven

---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:20|  ###############################################################################
2019-12-16T23:00:20|     XSIBACKUP-PRO 11.2.2: new execution request                     
2019-12-16T23:00:20|  ###############################################################################
2019-12-16T23:00:20|  
2019-12-16T23:00:28|  NOTICE: (c) XSIBackup kills any user launched jobs, make sure you don't overlap manual jobs
---------------------------------------------------------------------------------------------------------------------------------
XSIBackup PID:          1575299                                                 SocialServerJes
Mon, 16 Dec 2019 23:00:26 +0000                                    IPv4: X.X.X.X/255.255.0.0
VMware ESXi 6.0.0 build-5050593                              (c) Rsync 3.1.0 as opt. dependency
---------------------------------------------------------------------------------------------------------------------------------
Backup Id:                   01                       Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz 
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:33|  Backup description: backup all vms
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:34|  Message: crontab is installed for user root
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:35|  Backup user is: root
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:35|  Backup program is: xsitools
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:36|  /vmfs/volumes/Backup/xsit-repo is an XSITools repo
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:46|  --backup-point found at /vmfs/volumes/Backup/xsit-repo (filesystem: VMFS-5)
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:50|  The backup room has been limited to 1900 gb.
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:53|  Getting list of all VMs...
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:53|  12     VM1                [datastore1] VM1/VM1.vmx                             centos64Guest     vmx-11              
2019-12-16T23:00:54|  3      Mikrotik Firewall   [datastore1] Mikrotik Firewall/Mikrotik Firewall.vmx   otherLinuxGuest   vmx-08              
2019-12-16T23:00:55|  8      VM2              [datastore1] VM2/VM2.vmx                           centos64Guest     vmx-11              
2019-12-16T23:00:56|  9      VM3               [datastore1] VM3/VM3.vmx                           centos64Guest     vmx-11              
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:58|  VMs to backup:
---------------------------------------------------------------------------------------------------------------------------------
2019-12-16T23:00:59|  12     VM1                [datastore1] VM1/VM1.vmx                             centos64Guest     vmx-11              
---------------------------------------------------------------------------------------------------------------------------------
ERROR: cannot locate .ERR file at [/vmfs/volumes/datastore1/xsi-dir/.ERR-83d64b47e3d6103d7e158cc8b48a1c43c7097e1d]
2019-12-16T23:01:00|  [VM3] CLXSITO1 error: Killed
2019-12-16T23:01:00|  sh: bad number
2019-12-16T23:01:02|  Excluded disks removed from backup .vmx file

...

Offline

#2 2019-12-17 17:29:36

admin
Administrator
Registered: 2017-04-21
Posts: 1,386

Re: backup of VM failed with "ERROR: cannot locate .ERR file"

A "Killed" is being printed to STDOUT. Seems like you have some other processes running that are killing current one.
Run this command:

ps -c | grep xsi

And check whether you have multiple XSIBackup processes running in the background, if so kill them all and make sure your cron is not overlapping jobs.

Offline

#3 2019-12-18 09:10:21

lievenmoors
Member
Registered: 2018-11-21
Posts: 28

Re: backup of VM failed with "ERROR: cannot locate .ERR file"

Hi,

That was indeed the case. This was the output:

ps -c | grep xsi
2054591  2054591  sh                             /bin/sh /vmfs/volumes/datastore1/xsi-dir/xsibackup --backup-prog=xsitools:z --backup-point=/vmfs/volumes/Backup/xsit-repo --backup-type=Custom --backup-vms=xxx --backup-room=1900 --mail-to=x --use-smtp=1 --backup-how=Hot --backup-id=01 --description=backup all vms --exec=yes
2054592  2054592  sh                             /bin/sh /vmfs/volumes/datastore1/xsi-dir/xsibackup --backup-prog=xsitools:z --backup-point=/vmfs/volumes/Backup/xsit-repo --backup-type=Custom --backup-vms=xxx --backup-room=1900 --mail-to=x --use-smtp=1 --backup-how=Hot --backup-id=01 --description=backup all vms --exec=yes
2054598  2054598  grep                           grep xsi
1862327  1862327  sh                             /bin/sh -c "/vmfs/volumes/datastore1/xsi-dir/jobs/01"
1862330  1862330  busybox                        ash /vmfs/volumes/datastore1/xsi-dir/jobs/01
1862336  1862336  sh                             /bin/sh /vmfs/volumes/datastore1/xsi-dir/xsibackup --backup-prog=xsitools:z --backup-point=/vmfs/volumes/Backup/xsit-repo --backup-type=Custom --backup-vms=xxx --backup-room=1900 --mail-to=x --use-smtp=1 --backup-how=Hot --backup-id=01 --description=backup all vms --exec=yes
1965372  1965372  sh                             /bin/sh /vmfs/volumes/datastore1/xsi-dir/xsibackup --backup-prog=xsitools:z --backup-point=/vmfs/volumes/Backup/xsit-repo --backup-type=Custom --backup-vms=xxx --backup-room=1900 --mail-to=x --use-smtp=1 --backup-how=Hot --backup-id=01 --description=backup all vms --exec=yes

I have killed all those processes.
But I still don't know how they were spawned.
Do you have an idea?

Here is my cron configuration

cat conf/root-crontab
00 23 * * * "/vmfs/volumes/datastore1/xsi-dir/jobs/01"
ls /var/spool/cron/crontabs/
root
cat /var/spool/cron/crontabs/root 
#min hour day mon dow command
... (stuff not related to xsibackup)
00 23 * * * "/vmfs/volumes/datastore1/xsi-dir/jobs/01" # Added by XSIBackup
cat /etc/rc.local.d/local.sh 
#!/bin/sh

"/vmfs/volumes/datastore1/xsi-dir/src/cron-init" root
exit 0

Offline

#4 2019-12-18 10:58:36

admin
Administrator
Registered: 2017-04-21
Posts: 1,386

Re: backup of VM failed with "ERROR: cannot locate .ERR file"

Could be manual tests that were halted or launched by the cron. In your - ps -c | grep xsi - output the job seems to be the same for every spawned process and two of them have subsequent process Ids.

As we explain in the cron related posts, ESXi crond is a primitive cron binary included in busybox which lacks even a control script. It works well once you have configured everything as you want it to be and just leave it do its job, but the lack of any control may cause that you may launch multiple crond instances. Some people just do it to be sure that it's loaded in the belief that it will behave just like any other Linux crond service, which is not the case.

Also, be careful when doubling that zero, use a simple one, although that shouldn't matter much.

Offline

Board footer