©XSIBackup, Most common ©XSIBackup issues

 Please note that this post is relative to old deprecated software ©XSIBackup-Classic. Some facts herein contained may still be applicable to more recent versions though.

For new instalations please use new ©XSIBackup which is far more advanced than ©XSIBackup-Classic.

- Some users have reported issues related to broken pipes when using versions of ©ESXi that contained OpenSSH 7.9 like some builds of ©ESXi 6.5.0. You should patch your ©ESXi build to the latest available update and avoid OpenSSH 7.9 if possible. More details:

• VMWare Site
• Generic OpenSSH bug notes from Ubuntu site

Some users have reported a possible fix by overwriting the OpenSSH binary with some older one OpenSSH_7.7p1, OpenSSL 1.0.2r-fips. This may work, but could be wiped after a reboot. The recommended procedure is to patch the host, in case this would not solve the issue then a downgrade or upgrade to ©ESXi 6.7.0 containing a working OpenSSH version would be the best solution.

- Many users have reported that their HP servers yield a terribly slow throughput in data transfer. Many HP servers come with controllers that have their internal cache set to off, just turn it on in the corresponding BIOS page. Read this post for more details: enabling cache on an HP disk controller

- This is an annoying warning message thrown by OpenSSH version present in last versions of ©ESXi (6.7.0). We have addressed this issue by capturing the message in ©XSIBackup-Pro.

- If you are unsure whether some ghost or zombie processes are still running in the background in your ©ESXi host. This could happen if you have run some interactive backup sessions and have terminated them via Ctrl+C, or if you have overlapped some jobs. Just run this command in your ©ESXi shell to kill them.

OLDIFS=$IFS;IFS=";";for job in $(ps -c | grep xsi | awk '{print $1}' | \
sed -e ':a;N;$!ba;s/\n/;/g');do kill -9 $job 2>/dev/null;done;IFS=$OLDIFS

- Windows OSs can be sometimes tricky to configure. Here are the keys to comprehending what's involved in quiescing a Windows OS and avoid errors when taking snapshots.

- You will get this error if you use © copy;XSIDiff on a .vmdk file that is not in a VMFS volume. © copy;XSIDiff handles this and copies the file entirely, you can safely ignore this error.

- Failed to clone disk: The file specified is not a virtual disk (15).

You may get errors like the above when using NFS 4.1 in ©ESXi, switching back to using NFS 3 fixes the error. This is a compatibility problem between vmkfstools and NFS 4.1 implemented in ©ESXi.

You must understand that NFS 4.1 multipathing is a great asset in a multi-server setup, but perfectly useless to backup your VMs, as a single file cannot be transferred simultaneously through multiple NICs. So, use NFS 4.1 in your working environment if you will, but connect your backup devices using NFS 3. Of course, having a NAS device with multiple NICs will be greatly helpful, as you can connect different hosts to different NICs to load balance backups. A gigabit (still the most used speed specification) NIC can transfer up to 90 mb./s (theoretical limit). A HD with a decent amount of cache can go up to almost double that speed, and an SSD disk can hold four times as much, thus transferring through multiple NICs simultaneously is something very doable.

From ©XSIBackup 9.1.9 and up, the program will detect versions on both sides of the peer and choose an adequate Key Exchange Protocol (A.K.A. K.E.X.). Nevertheless, if you connect from an ©ESXi 6.5.0 box to a lower version one, the remote party will only offer KEX methods that the OpenSSH version present in ©ESXi 6.5.0 does not support any more, resulting in the message above. As a result, you can only perform IP data transfers from a lower version to a higher one or between same version, but not from ©ESXi 6.5.0 to a lower version of ©ESXi.

Having reverse DNS lookups enabled in your ©ESXi sshd daemon, can cause ©XSIBackup to delay backups or timeout without any apparent cause.

More info at: Fix reverse DNS issues

At Fujitsu, they built a buggy ©ESXi 6.0 version which ignores some basic ©ESXi design principles, like a persistent /etc/rc.local.d/local.sh file. They seem to have fixed it in their ©ESXi 6.5 ISO, but still, use the official ©ESXi release, unless you have no other choice.

More info at: https://forum.ts.fujitsu.com/forum/viewtopic.php?t=47837

This error/warning might be raised by OpenSSH when backing up big VMs by means of Rsync, which is tunneled through OpenSSH, after having copied some data. There are other issues related to Rsync getting killed by the SSH tunnel.

XSIBackup tunnels Rsync data through OpenSSH. Different ©ESXi builds might contain different OpenSSH versions. That message seems to be just a warning, but it does cut transmission inside the SSH tunnel in OpenSSH_7.3

https://marc.info/?l=openssh-unix-dev&m=147309628200704&w=2

Run this:

/usr/lib/vmware/openssh/bin/ssh -V

To determine your OpenSSH version and downgrade or upgrade your ©ESXi system for compatibility with ©XSIBackup if you plan to use Rsync. We will release in short an alternative program to substitute Rsync in OneDiff over IP. Nevertheless this renders OpenSSH 7.3 useless for some important tasks, so I guess ©VMWare will end up releasing a fix.

The CLI simply crashes when trying to switch on a VM backed up with XSIBACKUP-FREE. It may also crash if you imported the VM to ©ESXi 6.5 by other means.

This is caused by the .vmx key entry sched.swap.derivedName and seems to be a bug in ©ESXi 6.5.0. ©VMWare clearly states that the file and key are regenarated on every reboot.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006942

In previous versions of ©ESXi if that key was pointing to a wrong path or inexistent file, the system would handle it, in ©ESXi 6.5 if that path is wrong, the whole client crashes without further information.

WORKAROUND: just manually delete the sched.swap.derivedName entry in the .vmx file. XSIBackup-Pro addresses this issue automatically without the need of manual intervention.

You get the message "line 1: can't fork". This problem is a general bash issue and is caused by running out of available processes or memory. SOLUTION: reboot ©ESXi.

Keywords/ messages:

- VssSyncStart operation failed: IDispatch error #8449 (0x80042301)
- Create Snapshot: Create snapshot failed
- Unable to find a VM corresponding to ...
- Event 8, Volsnap. The flush and hold writes operation on colume C: timed out waiting for a release write command.
- The VSS service is shutting down due to an idle timeout

There are some issues related to ©VMWare Tools and Microsoft's VSS components.
First of all, make sure you don't have an obvious problem, check the following:

- That you have the latest version of ©VMWare Tools installed and that the service is running.
- The VSS Windows service is running and healthy.
- If your guest has SQL Server, Exchange, etc..., check that the VSS service for that program is running as well.
- The "Windows Virtual Disk" service is running and set to automatic start.

The following article might shed some light on the matter too:
TSM VM VSS error

It's a known issue:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2006849

Microsoft and ©VMWare blame each other. The sympthoms are:

- Duplicate snapshots files are created.
- The VM does not run on top of the newly created snapshot when it finishes to take it.
- Event IDs 50, 57, 137, 140, 157, or 12289 might be registered.
- No errors are thrown

This bug is critical in many ways, especially if you are using OneDiff with one of these operating systems. You can workaround by:

- Grouping this VMs into a backup job and using the --snapshot=dontquiesce argument. This will turn off quiescing.
- If you really need quiescing, use GPT partition style instead of MBR. This bug only affects Windows MBR systems.

Read this ©VMWare post for a workaround: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2068653

We have observed this behaviour very seldomly. It has to do with ©ESXi cron service. Other Linuses' crons suffer from similar problems also in rare occasions. It probably has to do with NTP and the clock slowing down. You can fix this issue by:

1/ Restarting the cron service

/bin/kill -9 $(cat /var/run/crond.pid) && /usr/lib/vmware/busybox/bin/busybox crond

2/ Reseting the time manually. The next command will reset the system clock and the hardware clock to 2017-06-17 23:57:00.

# esxcli system time set -d 17 -H 23 -m 57 -M 06 -y 2017 && esxcli hardware clock set -d 17 -H 23 -m 57 -M 06 -y 2017

3/ In case nothing of the above works reboot the ©ESXi host.

Most of the times, this error can be fixed by starting the [Virtual Disk] service in your Windows guest. Also take on account that MBR based Windows Servers cannot be quiesced due to an unresolved issue, read more

We have discovered that under some circumstances the cron service can keep on executing parts of code that remain loaded into memory. It is not clear what causes this erratic behaviour though. A reboot will fix this issue.

XSIBackup has a basic SMTP client, nevertheless it should work with the vast mayority of SMTP servers. Should you encounter any problem with your particular e-mail server, as ©XSIBackup is a script, you can easily tweak it to your needs. It's always far easier to simply use a different SMTP server though. ©XSIBackup supports gmail.com, yahoo.com and hotmail.com servers among others.

Keywords/ messages:

- 503 5.5.1 Error: send HELO/EHLO first: most of the times this error is raised because you didn't set the ©ESXi's hostname properly. Leaving the default "localhost" is not a valid entry.

Here you have two links where the restore config is explained via two different approaches, the latter is a lower level one, it offers more flexibility but may not work between different versions and/or hosts.

http://www.virtuallyghetto.com/2013/02/how-to-backup-restore-free-esxi-host.html

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2043048