#1 2018-08-02 15:50:59

toxeia
Member
Registered: 2018-07-18
Posts: 7

Appear to have two running jobs at once

Using the cron to run a job at 0430 UTC. Checked the ESXi cron to be sure there's no doubling, but it's just the one job like in the XSIbackup cron. I receive one email stating errors in the transfer, then another email about an hour later repeating the same error. The job is running OneDiff, hot, on 3 VMs. When looking at the log, I see that it's actually connecting to the SMTP server twice. The first time it's only completed the first VM, then it performs the backup for the 2nd and 3rd before sending another email with the same error. When checking the log, I see some lines are repeated as well.

Example from the log, just from the start of the job:

2018-08-02T00:30:04|  ---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:04|  ###############################################################################
2018-08-02T00:30:04|     XSIBACKUP-PRO 11.0.2: new execution request                     
2018-08-02T00:30:04|  ###############################################################################
2018-08-02T00:30:04|  
2018-08-02T00:30:04|  ###############################################################################
2018-08-02T00:30:04|     XSIBACKUP-PRO 11.0.2: new execution request                     
2018-08-02T00:30:04|  ###############################################################################
2018-08-02T00:30:04|  
2018-08-02T00:30:04|  NOTICE: (c) XSIBackup kills any user launched jobs, make sure you don't overlap manual jobs
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:04|  NOTICE: (c) XSIBackup kills any user launched jobs, make sure you don't overlap manual jobs
---------------------------------------------------------------------------------------------------------------------------------
XSIBackup PID:          4409260                                                 HPESXi.hba.msft
Thu, 02 Aug 2018 00:30:04 +0000                                 IPv4: 192.168.2.2/255.255.255.0
VMware ESXi 6.5.0 build-8294253                              (c) Rsync 3.1.0 as opt. dependency
---------------------------------------------------------------------------------------------------------------------------------
Backup Id:                  001                        Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz 
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:16|  Backup description: HBEsxi
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:16|  Message: crontab is installed for user root
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:17|  ADVICE: no SSD disks, please consider adding an SSD cache disk to improve performance
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:17|  Backup user is: root
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:17|  Backup program is: onediff
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:17|  Service OpenSSH ready at server 10.10.10.2:22
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:17|  Info: remote XSIBACKUP-PRO install dir has been set by means of the --remote-xsipath argument
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:18|  Remote xsi path set to: /vmfs/volumes/WDBlackR5/xsi-dir (filesystem: VMFS-6)
---------------------------------------------------------------------------------------------------------------------------------
XSIBackup PID:          4409264                                                 HPESXi.hba.msft
Thu, 02 Aug 2018 00:30:04 +0000                                 IPv4: 192.168.2.2/255.255.255.0
VMware ESXi 6.5.0 build-8294253                              (c) Rsync 3.1.0 as opt. dependency
---------------------------------------------------------------------------------------------------------------------------------
Backup Id:                  001                        Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz 
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:19|  Backup description: HBEsxi
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:19|  Message: crontab is installed for user root
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:19|  Remote ESXi version is 6.5.0
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:19|  Mirroring to server 10.10.10.2 port 22
2018-08-02T00:30:19|  Checking Rsync exists on the other side...
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:20|  ADVICE: no SSD disks, please consider adding an SSD cache disk to improve performance
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:20|  Backup user is: root
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:20|  Backup program is: onediff
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:20|  (c) Rsync (samba.org) found at [ 10.10.10.2:22:/vmfs/volumes/WDBlackR5/xsi-dir/bin/xsibackup-rs... ]
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:20|  Service OpenSSH ready at server 10.10.10.2:22
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:20|  Info: remote XSIBACKUP-PRO install dir has been set by means of the --remote-xsipath argument
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:21|  Remote xsi path set to: /vmfs/volumes/WDBlackR5/xsi-dir (filesystem: VMFS-6)
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:21|  (c) XSIDiff found at [ 10.10.10.2:22:/vmfs/volumes/WDBlackR5/xsi-dir/bin/xsidiff... ]
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:21|  Getting list of all VMs...
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:21|  6      HBSQL        [10kSASRaid5] HBSQL/HBSQL.vmx             windows9Server64Guest   vmx-13              
2018-08-02T00:30:22|  7      HBAD         [10kSASRaid5] HBAD/HBAD.vmx               windows9Server64Guest   vmx-13              
2018-08-02T00:30:22|  Remote ESXi version is 6.5.0
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:22|  Mirroring to server 10.10.10.2 port 22
2018-08-02T00:30:22|  Checking Rsync exists on the other side...
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:23|  8      Spiceworks   [10kSASRaid5] Spiceworks/Spiceworks.vmx   windows9_64Guest        vmx-13              
2018-08-02T00:30:23|  (c) Rsync (samba.org) found at [ 10.10.10.2:22:/vmfs/volumes/WDBlackR5/xsi-dir/bin/xsibackup-rs... ]
---------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:23|  VMs to backup:
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:23|  6      HBSQL        [10kSASRaid5] HBSQL/HBSQL.vmx             windows9Server64Guest   vmx-13              
2018-08-02T00:30:24|  (c) XSIDiff found at [ 10.10.10.2:22:/vmfs/volumes/WDBlackR5/xsi-dir/bin/xsidiff... ]
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:24|  Getting list of all VMs...
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:24|  6      HBSQL        [10kSASRaid5] HBSQL/HBSQL.vmx             windows9Server64Guest   vmx-13              
2018-08-02T00:30:25|  7      HBAD         [10kSASRaid5] HBAD/HBAD.vmx               windows9Server64Guest   vmx-13              
2018-08-02T00:30:25|  8      Spiceworks   [10kSASRaid5] Spiceworks/Spiceworks.vmx   windows9_64Guest        vmx-13              
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:26|  VMs to backup:
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:26|  6      HBSQL        [10kSASRaid5] HBSQL/HBSQL.vmx             windows9Server64Guest   vmx-13              
2018-08-02T00:30:26|  7      HBAD         [10kSASRaid5] HBAD/HBAD.vmx               windows9Server64Guest   vmx-13              
2018-08-02T00:30:29|  7      HBAD         [10kSASRaid5] HBAD/HBAD.vmx               windows9Server64Guest   vmx-13              
2018-08-02T00:30:29|  8      Spiceworks   [10kSASRaid5] Spiceworks/Spiceworks.vmx   windows9_64Guest        vmx-13              
---------------------------------------------------------------------------------------------------------------------------------
2018-08-02T00:30:31|  8      Spiceworks   [10kSASRaid5] Spiceworks/Spiceworks.vmx   windows9_64Guest        vmx-13  

You can see it gets the list of VMs and lists the ones ot be backed up twice - the second list of all VMs breaks up the list of VMs to be backed up.

Running the command manually doesn't create this doubling, nor does it lead to the errors. Likely my solution for now is just going to be using plink and windows scheduling to manage the backup schedule.

Offline

#2 2018-08-03 09:47:09

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: Appear to have two running jobs at once

Things don't happen twice magically. XSIBackup prevents overlapping cron execution requests, so this has all the appeareance that you launched a job through the cron and another one manually. Or that you just duplicated some cron line.

Just kill all jobs using the GUI tools and try again.

Offline

#3 2018-08-06 20:39:18

toxeia
Member
Registered: 2018-07-18
Posts: 7

Re: Appear to have two running jobs at once

I had previously killed all jobs before leaving the cron to take its course. After removing and reinstalling the cron again, it's still performing a double run.

My understanding is the lines in "(...)/xsi-dir/conf/root-crontab" is just inserted into the crontab located at "/var/spool/cron/crontabs/root" when ESXi boots. Is it possible there's another crontab somewhere that would be conflicting?

Offline

#4 2020-10-04 10:35:32

grudolf
Member
Registered: 2020-10-04
Posts: 1

Re: Appear to have two running jobs at once

Same situation here, we have a VMware ESXi 6.7 server where the cron jobs are executed twice:

2020-10-02T00:05:01Z crond[2099813]: crond: USER root pid 2471658 cmd /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
2020-10-02T00:05:01Z crond[2099813]: crond: USER root pid 2471659 cmd /etc/cim/lenovo/refresh.sh
2020-10-02T00:05:01Z crond[2099813]: crond: USER root pid 2471660 cmd /scratch/XSI/XSIBackup-Free/jobs/001 > /dev/null 2>&1
2020-10-02T00:05:01Z crond[2099698]: crond: USER root pid 2471664 cmd /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
2020-10-02T00:05:01Z crond[2099698]: crond: USER root pid 2471665 cmd /etc/cim/lenovo/refresh.sh
2020-10-02T00:05:01Z crond[2099698]: crond: USER root pid 2471666 cmd /scratch/XSI/XSIBackup-Free/jobs/001 > /dev/null 2>&1

It's a Lenovo server and another instance of cron is executed as /bin/crond from /etc/init.d/179.lenovo-cronjob or /etc/cim/lenovo/register_ipp_schema:

# ps -c | grep cron
2099698  2099698  crond                                         /bin/crond
2629914  2629914  busybox                                       /usr/lib/vmware/busybox/bin/busybox crond
#  ls -l /bin/crond
lrwxrwxrwx    1 root     root            35 Oct  3  2018 /bin/crond -> /usr/lib/vmware/busybox/bin/busybox

On server restart, the call to cron-init in /etc.rc.local.d/local.sh is needed to set up the contents of cron, but is also starts the busybox crond instance. Even if you kill the busybox crond process, xsibackup --update-cron will restart it.


Lenovo's mechanism to start/restart cron is

     CRONID=$(sed  '1q'    /var/run/crond.pid)
     kill -9  "${CRONID}"
     /bin/crond

but server seems to be reacting also to echo root >/var/spool/cron/crontabs/cron.update as a signal for crontab changes. Perhaps one of those could be used instead?

Offline

#5 2020-10-05 12:14:31

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: Appear to have two running jobs at once

If you are using some Lenovo build, switch to some VMWare official one, we don't support manufacturer builds, as they contain arbitrary changes with regards to VMWare's official build.

Offline

Board footer