©XSIBackup-App: how it works and file layout

Details on how backups work on ©XSIBackup-App appliance

Directory structure

©XSIBackup-App follows the same file structure as the latest versions of ©XSIBackup-DC. ©ESXi files are stored in the /scratch/XSI/XSIBackup-DC folder while Linux files are stored in the /home/[user]/.xsi folder.

You should respect the default locations, unless you know very well what you are doing, as ©XSIBackup will look for files in those folders by default. All files required to operate the ©XSIBackup server component are synced to the /home/[user]/.xsi directory automatically when you run some backup job against some Linux backend.

You may be wondering why we haven't respected the Linux FHS convention. There are very good reasons for that since ©XSIBackup allows to use different users than root, some of them with limited sets of privileges.

When you store a binary file to the /usr/bin folder you need to be root to set its permissions. If you aren't root you won't even be able to copy it there. When you are a non privileged user in a Linux server keeping binaries under the /home/[user] folder allows the [user] to use ©XSIBackup directly without requiring the root user intervention to set permissions or install software. The same principle is applied to storing the .ssh profile.

As said the required ©XSIBackup binaries are installed to /home/[user]/.xsi/bin when the installation is performed automatically by the client. The binary has execute permissions on those binaries and also write permissions on the whole /home/[user] folder, thus you can use any Linux server where you have a non privileged user and some storage in the /home/[user] folder.

SSL Keys

As you should know ©XSIBackup generates an RSA key pair to authenticate to backup hosts and storage back ends. The key pair is stored at the root of the installation directory for a given user. Thus the folders /home/root/.xsi, /home/alice/.xsi and /home/bob/.xsi would contain a key pair each.

The arguments --add-host and --add-key allow to put your .pub key to the remote server's authorized_keys file. Then you can use your private key to authenticate via SSH. Everything is automatic, the principles in regards to the file exchange are simple to understand though. You can delve into how keys work here at SSH.com.

The OpenSSH related files are stored at /root/.ssh for the root user and /home/[user]/.ssh for the rest of non privileged users when ©XSIBackup acts as a server. ©XSIBackup takes care to find the default location for every backup user and put the xsibackup_id_rsa.pub key to the corresponding authorized_keys file, creating it if it doesn't already exist.

There is one exception for the root user, as its home folder is set to /home/root instead of /root, this is to ease locating the storage volume and make it uniform with the rest of the users, which helps scripting backup or replica jobs.

Mount points

Anything that can be mounted to ©XSIBackup-App is mounted under the /mnt/XSI dir. That includes backup servers, NFS, iSCSI, SSHFS and Samba mount points, etc...

It is indeed important that you respect the default mount point locations when working with ©XSIBackup-App, as the GUI per instance will look for files and folders attending to this structure. Thus, if you don't want to do extra work debugging and setting things up, we recommend that you leave the default locations.

The structure of the /mnt/XSI directory looks something like this:

Ultimately you should place all mount points under the /mnt/XSI dir classified by transport protocol, except the /mnt/XSI/srvs directory, which should contain the servers to be backed up, in other words, the servers that contain the data to be backed up. In this case and given the notoriety of this set of servers we assign them a directory.

The installation dir file structure

/home/[user]/.xsi/bin

This dir contains the binaries that allow the server to work.

xsibackup
The most important binary is xsibackup, which is an exact copy of the binary you will find at the client's root installation dir.

sendmail
This is the bash script that contains the SMTP client in ©XSIBackup-DC, the ©ESXi's backup service version of our software that runs directly in the ©ESXi hypervisor. It uses STARTTLS method to send encrypted e-mails, you should point your server to port 587 when using it.

sendcurl
This is a new bash script to send e-mail from ©XSIBackup-App (default SMTP client) using curl which comes preinstalled by default in most Linux distros. This SMTP client uses a direct TLS connection to the SMTP host as opposed to sendmail above which uses STARTTLS method. Point the SMTP server to the TLS port, typically 465.

sort
This is a copy of the well known Linux sort binary it is used to sort data (block manifests and .map files). We use our own version of this binary to prevent differences in the implementation of the arguments that may cause errors otherwise. Using a custom version ensures that the accepted arguments will not cause trouble. The client side uses GlibC built in functions to sort data, on the server side we have kept sort as an external dependency, it works well and is fast for the task.

xsib64
Same as with sort. We use this base64 encoding binary to ensure that the remote end is able to decode base64 encoded strings. We use it mainly to decode remote bash commands sent from the client. Sending a base64 string eliminates any issues related to escaping especial characters and ensures a smooth user experience in regards to that.

/home/[user]/.xsi/etc

xsibackup.conf
This file contains different default values used by the xsibackup binary. You should not edit this file unless directed by our support team.

smtpsrvs.conf
This contains the data for the SMTP servers used by ©XSIBackup to send the backup reports. The structure of the file is briefly explained inside of it. As you may imagine it contains: smtp server, port, username, password, etc...

The jobs folder
This directory contains the backup and replica jobs created from the GUI or saved from the command line by appending the --save-job argument. The backup job files are regular bash files that contain the call to the xsibackup binary along with the arguments configured for the job. Everything is redirected to the var/log/xsibackup.log file in each file.

scripts
The job folder contains a subfolder called scripts which is there to hold the scripts called by the --prev-script and --post-script arguments when working with Linux servers as the source of the backup data.

/home/[user]/.xsi/gui

This is the folder containing the nCurses GUI. The gui is dependent on the dialog binary, which is a classic Linux nCurses library used to generate basic Graphical User Interfaces.

main
This file initializes and loads the GUI.

src The src dir contains the GUI components organized in modules. We won't enter into more details here. Everything in the GUI is a bash script thus you can read the files to get all the details.

dat The dat dir contains the data the GUI needs to operate like the list of program's arguments and possible values, it also holds some temporary data.

/home/[user]/.xsi/lic

©XSIBackup uses the lic folder to store the keys for each of the licensed servers. It first stores the MAC_request.key files that you can upload to your user panel to get a license key. You can place the license keys as obtained from the license server in this folder and they should work. Since latest versions you can install your license keys automatically from the GUI provided that you have a valid client login.

/home/[user]/.xsi/scripts

The scripts folder contains two scripts:

reset-appliance.sh
This script reinitializes the appliance. It deletes all user preferences and mount points.

xsibackup-update.sh This is the script called by the xsi-update command. It will look for a newer version of itself at Sourceforge.net it will update itself if it finds a newer version. Then it will look for newer versions of the appliance's software at 33hops.com and update all the software after asking for confirmation. You need a client Id to upgrade the appliance on line. Unlicensed users can download a newer version of the appliance.

/home/[user]/.xsi/tmp

This is one of the 2 temp folders that can be used, the other one is the main temp system directory at /tmp. You can delete the contents in this folder, should they accumulate with time, do so when there isn't any job running. There is a section in the GUI that allows to empty the tmp folders.

The temp folders location can be set at the xsibackup.conf file. It is strongly recommended that you keep the default values though.

/home/[user]/.xsi/var

The /var directory holds the logs and templates that are required to operate the software.

General Linux dependencies

©XSIBackup-App comes with everything installed and self contained, so that you don't have to install all the software and its dependencies. We may prepare a Linux installation package in the future, but this is not one of our priorities. The reason is that every sysadmin has his own favourite distro and different Linux versions and distros can contain substantial differences that would make support a much more difficult task.

Don't try to extract the binary from the appliance to install it on other Linuses, it is unsupported for very good reasons, and we will not attend support requests regarding this topic.

Not would only offering support for a vast amount of many different Linux distros become a nightmare for us, but we have found that serious critical bugs aren't that uncommon even in well known Linux distros. By offering the ©XSIBackup-App appliance as a software operational unit we are saving you the time and pain of discovering them on your own.

On the other side each distro seems to be focused on some particular issue, like one of the best: Debian, which is very much security oriented. This has made their developers to deprecate old system calls like vsyscall that ©XSIBackup-App still uses and to turn services by default like iNotify that can become a nightmare on a slightly loaded host, as it can really affect performance.

We will mention now some of the dependencies for the xsibackup binary:

swaks
This is a well known command line SMTP client. It works well with most servers, it won't allow you to use open relays inside your LAN though as oposed to the old sendmail bash script which was more permissive in regards to that.

FUSE
File System in User Space system for Linux, allows some protocols to work, like SSHFS as well as our propietary XSIFS file system to mount backup repositories.

NTFS-3G
Driver for the well known Windows file system. Required to mount and access Windows .vmdk virtual disks.

Samba
The Linux CIFS compatible file sharing protocol. Not strictly required, unless you want to mount remote CIFS shares, which is not recommended, as NTFS nor CIFS will not offer you a performance level close to that of XFS or ext4.

Dialog
The classic nCurses Linux library.