Last updated on Monday 28th of February 2022 08:52:48 PM

©XSIBackup: troubleshooting the cron service

Procedure to check the ESXi cron service and put it to work

XSIBackup Crontab

This post applies to both the Free and Pro versions of the software. The only difference in regards to the cron facility is that in the Pro version you have the option to enter the schedules in a window, instead having to edit text files with vi. No big deal in regards to how things work, apart from having to pay extra attention to how you write cron lines.

Up until version 11.0.0 XSIBackup used a crontab named xsibackup-cron which was placed in the root xsi-dir installation directory. This worked well for some time and users got used to it. Nevertheless, it was an abstraction created not to have to deal with the ESXi crontab, which is not persistent and on top of that is a very basic piece of code which lacks control code and a proper service init system. As a result, the possibilities it offered were limited to scheduling weekly backups, and setting up a daily backup would require generating a mask for each of the seven days.

With the upcoming of version 11.0.0 we have solved this limitations by saving the schedules in a crontab that's now named root-crontab and is placed inside the /conf folder, relative to the root installation directory. The contents of this file are just copied over to the /var/spool/cron/crontabs/root when the system reboots, or when you invoke the --update-cron argument.

The new system allows to employ classic Linux/ Unix crontab syntax, so now, there isn't any constraint in the way you may program your schedules. You can have daily, weekly, monthly, etc... schedules as per the ESXi Linux crontab, which is bundled in the busybox binary shipped with ESXi.

Known issues:

The Day Of Week column in ©ESXi may need to be passed as the day abbreviation: mon, tue, wed, thu, fry, sat, sun. If we say "may", it is because every ESXi version has a different busybox implementation and the cron daemon implementation it contains is not like the most sophisticated ones you can find in advanced Linux distros. So try to keep your schedules simple and should you need to use the DoW column, try different numerical formats and also abbreviations.

Troubleshooting:

Add > /dev/null 2>&1 at the end of backup jobs to redirect everything to /dev/null. As backup job output is previously redirected to the xsibackup.log file all output text will be stored there, doing this can prevent some awkward behaviours from part of the shell.

0 2 * * * "/vmfs/volumes/datastore1/xsi-dir/jobs/001" > /dev/null 2>&1


XSIBackup takes care of checking the crond binary is loaded just once and that the contents of the /conf/root-crontab file are copied to the ESXI crontab when using the --update-cron argument, so the chances the ESXi crond binary hangs are very little if you just use XSIBackup commands and do not deal with the ESXi command line directly.

If you can't get your XSIBackup job to be fired, there is some procedure that you should follow. Follow it blindly, if you go your own way we cannot predict the results. The matter is fairly simple:
1/ Make sure your backup job is working. Call it by means of the --run-backup argument, or just execute the job by calling it with an absolute path, i.e: /vmfs/volumes/datastore1/xsi-dir/jobs/001 and run tail -f /vmfs/volumes/datastore1/xsi-dir/conf/xsibackup.log in a different window.

Do not try to reinvent the wheel, compose your backup jobs as stated in the different available posts and documents: xsibackup-free-11-things-that-have-changed.html

2/ Once you know the above is O.K., not before; reset the pid of the ESXi crond process.

for p in $(ps -c | grep -v grep | grep 'busybox crond' | awk '{print $1}'); \
do kill -9 $p; \
done; \
/usr/lib/vmware/busybox/bin/busybox crond; \
ps -c | grep -v grep | grep 'busybox crond'


Should return:

33261 33261 busybox /usr/lib/vmware/busybox/bin/busybox crond


The above command should return a line with the crond PID as shown. Should you get something different, then that means you committed a mistake when cutting and pasting, your ESXi command line is not working, or you chose the wrong SSH terminal window.

3/ Finally, we are going to make sure the cron service is running. To achieve that, we'll simply write the date to a test file directly from the /var/spool/cron/crontabs/root file. To edit this file, you first need to make sure it is indeed editable by the root user, so run:

chmod 0700 /var/spool/cron/crontabs/root


Now edit it with vi and add

*/1 * * * * echo "$(date)" >> /tmp/my-cron-test.txt


From a different terminal window, just tail -f the file

tail -f /tmp/my-cron-test.txt


You should now see a new line being written to the /tmp/my-cron-test.txt every minute. When you are done testing just run the code below to reset the permissions on the root crontab.

chmod 0600 /var/spool/cron/crontabs/root


Now, if you have a working cron daemon and a working backup job, the only chances that your jobs are not being executed, is that you are using a wrong path somewhere.

If you don't have a working cron daemon, then your ESXi installation is broken, or you missed some step above. A loaded crond daemon must write the date to the test file as the root user is "all mighty" and the /tmp folder allows everyone to write there.

There are always chances that you did something wrong: edited the cron file with an editor that changed the page code, so the cron daemon is not able to read it, misplaced some element in the cron file, etc... If that is your case, you don't know what to do and you don't comprehend some of the facts involved, then just reinstall ESXi from scratch and follow this troubleshooting guide editing the files with vi. If you don't know how vi (AKA vim) works, then read this guide: https://www.howtogeek.com/102468/a-beginners-guide-to-editing-text-files-with-vi/

We all had to learn, sometimes the hard way. If you pretend to use XSIBackup, you need to have or acquire some basic Linux command line skills. Our support department is not there to take your hand and teach you the basics.

Always check your log to see what the XSIBackup process may have written to it in search of errors and warnings. Change the number of lines (-n20) to suit your needs.

tail -n20 /vmfs/volumes/datastore1/xsi-dir/conf/xsibackup.log


Daniel J. García Fidalgo