©XSITools: how to speed up data throughput

Please note that this post is relative to old deprecated software ©XSIBackup-Classic. Some facts herein contained may still be applicable to more recent versions though.

For new instalations please use new ©XSIBackup which is far more advanced than ©XSIBackup-Classic.

This approach is only possible in ESXi 5.5 and below. Since ESXi 6.0 VMWare cut down the possibility to create socket files, thus Controlmaster Multiplexing will not work in ESXi 6.0.0 and above.

©XSITools tunnels chunked data through SSH. Every time an SSH data transfer is performed, the SSH authentication and handshaking takes place. This takes from 0.3 to 0.7 seconds for each block transfer and can represent up to 10-25% of the time it takes to upload your data.

Fortunately, the SSH server includes a feature to allow reusing sockets for subsequent connections, it is called ControlMaster, and can be used to speed up your data throughput when using ©XSITools in your backups.

Many of our clients, which are familiar with the SSH protocol already use this directive. Activating it is a fairly trivial task, but we have decided to write this post to shed light on this matter and encourage users which are not so familiarized with the use of the SSH protocol to activate it in their hosts.

By enabling the ControlMaster in your SSH host, you will reduce the connection establishment period to 0.01-0.02 seconds, which will make your ©XSITools backups to be much faster.

We will probably be adding this feature to next main version of ©XSIBackup, so if you are using version 11.3.0 or above, you will most probably not need to activate this by hand.

If you have some older version and you want to enable this feature, you just have to edit the main xsibackup file, look for the SSHOPTS variable and set that line to:

SSHOPTS="-o PreferredAuthentications=publickey -o StrictHostKeyChecking=no -i \"${PWD}/xsibackup_id_rsa\" -o controlmaster=auto -o controlpath="/tmp/ssh-%r@%h:%p" -o ControlPersist=30s"

If you still feel not very confident about editing the file yourself, you may just execute this script, which will patch your xsibackup file. Please, note that this script will look for the first xsibackup file in your system and patch it, if you have multiple xsibackup files in your server, only the first will be patched.

Don't forget to backup your xsi-dir first. This script copies your xsibackup file to the /tmp dir, but you should always cover yourself from loosing data.

Cut & paste the below command to your SSH client, wait some seconds while the code is parsed and press enter, you should see a notice stating: "Patch successfully installed -- OK".

Parsing multiline scripts by cut&paste can sometimes return some syntax error, at least in Putty. If that is your case you may retry, we found that it's always parsed correctly when you insist. In our case, Putty deletes some new line characters randomly for some unknown reason.

You can always paste it into a script file and run it, which will always be a winner option. The script may take some seconds to find your xsibackup file, it will look up to 5 directories under /vmfs/volumes.

xsibackup_path="$( find /vmfs/volumes -maxdepth 5 -name xsibackup -exec ls -1 {} \; | head -n1 )";
cp ${xsibackup_path} /tmp;
sed -i "s/^SSHOPTS=\"-o PreferredAuthentications=publickey.*/SSHOPTS=\"-o \
PreferredAuthentications=publickey -o StrictHostKeyChecking=no -i \
\\\\\"\${PWD}\/xsibackup_id_rsa\\\\\" -o controlmaster=auto -o \
controlpath=\\\\\"\/tmp\/ssh-%r@%h:%p\\\\\" -o ControlPersist=30s\"/" \
"${xsibackup_path}" && ( clear && echo -e "----------------------------------\n\
Patch successfully installed -- \033[0;32mOK\033[0m\n----------------------------------" ) || \
(clear && echo -e "----------------------------------\nError applying the patch -- \
\033[0;31mKO\033[0m\n----------------------------------" )

You can also check the difference on your own

Switch to your xsi-dir folder and run this command against any previously "linked" server. Replace the value of the SERVER_IP and SERVER_PO variables by your linked server's IP and port.

Note: A linked server is a host to which you have added XSIBackup's public key: xsibackup_id_rsa.pub to its authorized_keys file. This is a general SSH procedure which you may easily perform by using the built in --link-srv argument.

SERVER_IP="192.168.40.100"
SERVER_PO="22"
time ssh -p${SERVER_PO} root@${SERVER_IP} -o controlmaster=auto -o controlpath="/tmp/ssh-%%r@%%h:%%p" -o ControlPersist=30s -o PreferredAuthentications=publickey -o StrictHostKeyChecking=no -i xsibackup_id_rsa "echo 'HELLO WORLD!'"

The controlpath option is at the same time including some variables wich refer to the remote user name (%r) the host (%h) and the port (%p). Note that the socket will be reused up to the number of times defined by the MaxSessions variable in the server configuration file (/etc/ssh/sshd_config). MaxSessions defaults to 10, so you may increase it manually to stretch the benefits of Controlmaster.

Note that we have added the time command at the beggining of the check. First time that you execute the below command, it will take around 0.5 seconds, depending on your network speed and whether you are running the check in a LAN or through a VPN, a WAN or an Internet connection.

When you run the check on subsequent runs, before the socket timeout (ControlPersist=30s) expires. You will see the command is run much faster. If you let the socket timeout expire, you will notice that the first additional run returns back to around 500 milliseconds. The same will happen when the MaxSessions variable limit is reached.

As an excerpt, adding the Controlmaster directive to the SSH Client command list options is enough for the client to create a socket that it will reuse on subsequent SSH connections. Be aware that this option is only available from OpenSSH 5.6 and above.

Daniel J. García Fidalgo
33HOPS