©ESXi Cross Version Backup

 Please note that this post is relative to old deprecated software ©XSIBackup-Classic. Some facts herein contained may still be applicable to more recent versions though.

For new instalations please use new ©XSIBackup which is far more advanced than ©XSIBackup-Classic.

Recent ©ESXi versions can use one of two VMFS versions: 5.0 & 6.0. Version 6.0 adds some new features, like: SE Sparse format as default (this was previously only used when disks were bigger than 2 tb.) and addresses some problems related to aligning the guest OS block size to the way data is hadled by the virtualization file system. The main issue that has been resolved by means of SE Sparse format is the ability to reclaim unused space, or blocks that are deleted in the guest OSs and can be made available for provisioning again, this was not possible in VMFS5.

VMFS versions are not compatible with each other. You cannot, per instance, use ©OneDiff to clone a VM from a VMFS5 datastore to a VMFS6 one, neither the other way around. This is due to ©OneDiff using snapshots to exchange data; thus, the snapshot would need to be valid in a different VMFS environment, which is not the case.

You can perform other type of backups: Vmkfstools, Rsync or ©XSITools backups can be done from a VMFS5 to a VMFS6 datastore and viceversa. However, you might not be able to use them in the new location. This doesn't have to do with the VMFS version itself, but with some other concern that we'll treat next: the Hardware Version.

This is the second thing to care about when mixing ©ESXi versions and moving VMs around. Just like hardware evolves in the real world (although real is an increasingly relative term these days), Virtual Hardware does as well in the ©VMWare world. Hardware emulation is updated with new features every time a new ©ESXi version is launched.

You are, most of the times, given the chance to keep your HW version when movinmg a pre-existing VM from one host to a new one. This is, most of the times, the wiser decision, especially in mixed environments, although it's not the only reason why you should do it.

When you work in a mixed environment you should keep the HWV that the eldest ©ESXi version can run. This will ensure that when you move Virtual Machines around the hosts, of course under a uniform VMFS version, the different hosts can run the VMs without problems.

Should you upgrade some Virtual Machine hardware version, you will lose its capacity to be used in previous versions of ©ESXi.

Take a look at this ©VMWare post to see the exact compatibility list:
Hardware Version Compatibility Chart

Another important thing to take on account when using an environment that includes mixed ©ESXi versions, is the OpenSSH versions shipped with each ©ESXi system and whether they are compatible with each other. You will never have problems by running local backups by means of Vmkfstools or ©XSITools , but exchanging data over an SSH tunnel demands some attention and knowledge from your part.

Of course, you can always ease things by using the same ©ESXi versions, that way you will ensure that no compatibility issues arises at the SSH tunnelling level when moving data around.

Even if you backup from a lower ©ESXi version, let's say 5.5.0 to a higher one, i.e.: 6.5.0, you will still be in a fairly manageable scenario. But if you pretend to backup from ©ESXi 6.5 or 6.7 to ©ESXi 5.1 or 5.5, you will be playing in the honour league, so you'd better be prepared for the game, of course, you might even not be able to accomplish what you want.

OpenSSH consists in two programs:

1/ The client, normally known as ssh and invoked by the same name in the command line. The client's configuration file is usually found at /etc/ssh/ssh_config, but you may very well find none, as all client parameters can be passed in the command line. In fact ©XSIBackup passes all SSH client configuration parameters that way, it builds them dynamically depending on your backup job configuration parameters.

2/ The SSH Server AKA: SSH Daemon, or SSHD binary. It's configuration file can be found at /etc/ssh/sshd_config. It contains options used to tell the SSHD daemon how to handle different aspects of its functioning at run time. This file is parsed every time the server launches a process, thus you usually don't need to restart the SSHD daemon.

There are basically two things to take on account when connecting two SSH systems:

1 - KEX (Key Exchange Protocol):

This is the most important of the two, basically because there have been changes in the last years and OpenSSH versions shipped with ©ESXi. One of the most widely used KEX protocols in the past has been Diffie-Hellman and its multiple variants, but it was deprecated time ago due to some security concerns. Thus, depending on your ©ESXi or Linux version and in turn the OpenSSH version shipped with it, you may find that it is allowed by the SSH server, or not.

Rsync, which we include in XSIBackup, is version 3.1.0 without any modification. We haven't touched it to keep its Open Source license immaculate and not to have to deal with the derived additional work and/or legal concerns. So, it still tries to use some Diffie-Hellman KEX algorithm when tunnelled through SSH, when it is not able due to the receiving part rejecting it, it launches a warning, which is the infamous: kex protocol error. We have simply captured and discarded this message from XSIBackup-Pro version 11.2.3, as in the end it's an irrelevant piece of information that just tends to misguide users.

The KEX algorithm can be controlled with the -o KexAlgorithms=[some algorithms] option from the SSH client. and with the KexAlgorithms key in the server side. The following is an example enabling different KEX algorithms in the /etc/ssh/sshd_config on the server side.

KexAlgorithms ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256

We have configured ©XSIBackup to always try some common KEX algorithm denominator when connecting to some remote system that is below the initiating SSH version, but it's just a matter of time that all offered KEX algorithms are rejected, so try to keep an homogeneous ©ESXi pool of servers without too distant versions.

2 - Ciphers

Ciphers are the algorithms used to encrypt data on the SSH tunnel. They have been gradually strengthened over time in new OpenSSH versions, and some of them have also been deprecated for security reasons.

Current OpenSSH default Ciphers are strong, but they produce a high CPU load, which in turn causes the data transfers to be slower than they could. That's why from XSIBackup-Pro 11.2.3 we have enabled a new feature called Less Secure Ciphers. You should only use them after assuming that they are not secure any more.

The reason for enabling this feature, that is activated through the [l] switch in the --backup-prog argument is to allow Sysadmins to increase the speed of their backups when they have some additional security measures that make re-encrypting unnecessary, or when encryption is simply not needed or wanted.