Last updated on Sunday 16th of July 2023 03:17:27 PM

Azure off-site ©VMWare ©vSphere Backup

Backup your ©ESXi virtual machines to a Linux Azure virtual machine with ©XSIBackup-App

We don't have to remind you that a clean uncontaminated backup is the only way to protect your data from modern threats, such as ransomware targeting ©VMWare ©ESXi servers.

Virtual machine off-site backup Virtual machine off-site backup

The next though that will come to your mind is whether it's a good idea to keep your backups along with the rest of your enterprise data.

You not only want to protect your virtual machines and data in general against ransomware and viruses. The rest of threats still exist: thefts, fires, flooding etc.... All of the latter can ruin your backups, should you keep them in some on-site server.

That's why some form of off-site or cloud backup is a must.

There is a plethora of public cloud services, VPS, dedicated bare metal servers, etc... these days. What should we choose?.

Today we are going to see how to use a Linux Azure VM to store your deduplicated backups. You could also store replicas, but that wouldn't make much sense, given that you would not be able to mount them reliably in some on-premises ©ESXi host.

Azure Linux VM as a backup backend

Azure is indeed an interesting cloud service provider. If you compare it to AWS you will soon find some weak points in comparison, but it has a really interesting feature: it is really user friendly, intuitive and you can deploy things much faster than in AWS, which is in comparison, cryptic, difficult to use and understand and certainly not user friendly when compared to Azure.

Azure allows to deploy Linux VMs quickly. For our tests we used Ubuntu, although any other Linux compliant distro should work too. Please do mind each distro's characteristics before using them. Per instance Debian will enable iNotify service by default and disables vsyscall system call which will force you to deal with those issues before using it.

In regards to sizing the VM, just 2 cores and 2 GB of memory will do it. Adding more CPUs and memory will not improve performance, but will increase your bill.

You will need to add some disk to your Ubuntu VM to act as the storage container to your backups. We uaed a 200GB Hard Disk for our case study.

You will have to pay for the disk holding the backup volume, you won't need the VM to be on all the time, thus you can take advantage of the possibility to switch the VM on only during the backup windows. There are several ways to accomplish that and we won't cover that in this post.

How to create and configure the Azure Linux VM

To create a new Ubuntu VM go to Virtual Machines from the home page of Azure, then click on "Create Azure Virtual Machine".

Azure: create Virtual Machine Azure: create Virtual Machine

You will have to fill the details of your new VM in. We have chosen Ubuntu 20.04 in a VM with 2 vCPUs and 8GB of RAM. We couldn't find anything smaller, in your case, as previously stated 2 vCPUs + 2 GB of RAM are enough.

Please, note how we have chosen some data center in France (the closest we could find in the list). Our test lab is in Madrid (Spain), thus we want the lowest possible network latency to maximize our data throughput. In our case we can get 20-25 ms, always choose the closest datacenter to your location, you should be able to reach better figures in many places.

In case the latency to your closest Azure datacenter is above 50-60ms, it might not be worth for you to try to backup a big amount of data, as the network latency will reduce the effective bandwidth to 6-7 MB/s. Check whether you can find some local provider or set a backup server at home.

Another interesting feature is Azure Spot Discount, which will drastically reduce your bill by using infrastructure leftovers. This isn't a good strategy for some busy production server, but could be a perfect fit for backups, as they are usually performed during out of office hours.

Azure: fill in the details of your Virtual Machine Azure: fill in the details of your Virtual Machine

And now the security related matters. For this case study we have chosen to authenticate via username and password. Azure will allocate a privileged user that we have called backup01. By using this user name we will later add an RSA key to the backup host to be able to use it as a backup backend.

All traffic will be tunnelled through a single TCP port, SSH which default port number is 22. We can easily filter the traffic to only allow access from our fixed office IPs, or leave it open as in this case. This is not the recommended procedure, but given the fact that our case study won't last long and that the VM will be off most of the time, it's fair enough for us.

Azure: fill in the details of your Virtual Machine Azure: fill in the details of your Virtual Machine

Once you have completed filling the details of your backup Linux Ubuntu 20 Azure virtual machine you can create it. Now you can switch the VM on and login using your favourite SSH client and the backup01 username on port 22 of the public IP. You can find the public IP in the main details of your newly created VM.

Azure creates a VM with a private IP in a network and sets a NAT from the public IP, which is very convenient to avoid exposing unnecessary ports to the outside world.

Once you login with the backup01 username you can either run commands directly with it or sudo to run commands as root. You will have to keep in mind that if you, per instance, create a directory using sudo, the owner of that dir will be root, so you would need to run sudo chown -R backup01:backup01 /home/backup01 to make backup01 the ownser of the dir structure, otherwise you may have permission errors when trying to copy data as backup01, but we'll come back on it later.

The backup disks

The VM we just created is barely 20 GB in size, thus it won't be able to store much data. We need a backup volume to where we can save our valuable backups. We will also need that volume to be persistent across reboots, so that we don't lose our backup data when we reboot the VM.

Pay special attention when using some Azure free account, as the contents of your disks may be wiped upon a reboot of the virtual machine.

Click on disks and then on Create and attach a new disk. A new disk will be added to the list of disks.

You should choose some Standard HDD, as the extra speed of an SSD will be useless when our bottleneck is the network throughput and they are indeed cheaper.

Azure: adding a disk to your Virtual Machine Azure: adding a disk to your Virtual Machine

Once the disk has been added to your VM you can list the available disks in the virtual machine by running lsblk.

backup01@LBAK01:~$ lsblk
loop0 7:0 0 63.5M 1 loop /snap/core20/1950
loop1 7:1 0 63.5M 1 loop /snap/core20/1974
loop2 7:2 0 91.9M 1 loop /snap/lxd/24061
loop3 7:3 0 53.3M 1 loop /snap/snapd/19457
sda 8:0 0 200G 0 disk
sdb 8:16 0 30G 0 disk
├─sdb1 8:17 0 29.9G 0 part /
├─sdb14 8:30 0 4M 0 part
└─sdb15 8:31 0 106M 0 part /boot/efi
sdc 8:32 0 16G 0 disk
└─sdc1 8:33 0 16G 0 part /mnt

As you can see in the upper output a new 200GB disk has been detected in the Ubuntu 20.04 virtual machine as /dev/sda. Now we have to format it and mount it. You could also use the classic fdisk tool for such a small disk, but you should get used to employ parted, which allows to partition bigger new disks over 2TB.

We invoke parted via sudo with admin rights with our new disk /dev/sda as the first and only argument and set a gpt label. We then quit and run a one liner (sudo parted -a optimal /dev/sda mkpart primary 0% 100%) to generate a primary partition using 100% of the disk.

backup01@LBAK01:~$ sudo parted /dev/sda
GNU Parted 3.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel
New disk label type? gpt
(parted) quit
Information: You may need to update /etc/fstab.

backup01@LBAK01:~$ sudo parted -a optimal /dev/sda mkpart primary 0% 100%
Information: You may need to update /etc/fstab.

backup01@LBAK01:~$ sudo mkfs.xfs /dev/sda1

Finally we format it with mkfs setting an XFS file system. To mount the disk we just have to create a mount point, in our case it is /home/backup01/backups

Depending on the distro you use you might be able to create the dir without using sudo, as it is inside the home folder of the user we are logged in as: backup01. If you receive some permission error use sudo mkdir /home/backup01/backups and remember to reset the permissions back to backup01 by running:

chwon -R /home/backup01

To mount the dir you just have to run:

mount /dev/sda1 /home/backup01/backups

To make the mount point permanent add the following line to /etc/fstab

/dev/sda1 /home/backup01/backups xfs defaults 0 0

Now you have a backup server ready in the Azure cloud ready to listen on port 22 of public IP a.b.c.d

Backing up your ©VMWare ©ESXi virtual machines to the Azure Linux server

In this case study we are going to use our software ©XSIBackup-App. It is an appliance based in Linux CentOS 7 that contains ©XSIBackup 2.0 binaries as well as an nCurses GUI that will allow to manage the backup of multiple attached ©ESXi servers from one single console.

To start backing up our VMs there, we just need to exchange ©XSIBackup's public key, present at the root of the installation dir. We don't even have to reference it, as ©XSIBackup will use it by default.

Run this command from ©XSIBackup-App command line or use the Link menu in the GUI and enter the password for the backup01 user when prompted to.

xsibackup --add-key backup01@a.b.c.d:22

You will just have to enter the password once. You will get the output below:

[root@XSIBackup-App ~]# xsibackup --add-key backup01@a.b.c.d:22
System Information: Linux, Kernel 3 Major 10 Minor 0 Patch 1160
PID: 1804, Running job as: root
Key exchange routine
This is the routine to exchange keys with other servers
You will be prompted to enter the remote server's password
a number of times. Please do so, this is a one time operation
backup01@a.b.c.d's password:
The remote server is of type: linux OK
Let's look for the 'authorized_keys' file there...
The /home/backup01/.ssh/authorized_keys file does not exist
Do you want to create it? (y/n): y
The file /home/backup01/.ssh/authorized_keys was created
Checking if local public key exists at remote file...
Add local public key to backup01@a.b.c.d:22:/home/backup01/.ssh/authorized_keys?
Local public key <xsibackup_id_rsa> has been
added to backup01@a.b.c.d:22:/home/backup01/.ssh/authorized_keys OK
Remote server successfully linked for user backup01
SSH session was closed OK
Removed PID OK

(*) Please, note that we changed the real public IP by a.b.c.d

From this moment ©XSIBackup-App is linked to the remote backup server and we will be able to launch backup jobs as user backup01.

We should make sure that the folders under /home/backup01 are indeed owned by the backup01 user, otherwise we won't be able to run our jobs sucessfully and will receive a permissions error when launching the job.

If you list the contents of the /home/backup01 home folder, like this.

backup01@LBAK01:~$ ls -la /home/backup01/
total 40
drwxr-xr-x 6 backup01 backup01 4096 Jul 16 11:02 .
drwxr-xr-x 3 root root 4096 Jul 12 16:50 ..
-rw------- 1 backup01 backup01 6 Jul 12 16:54 .bash_history
-rw-r--r-- 1 backup01 backup01 220 Feb 25 2020 .bash_logout
-rw-r--r-- 1 backup01 backup01 3771 Feb 25 2020 .bashrc
drwx------ 2 backup01 backup01 4096 Jul 12 16:52 .cache
-rw-r--r-- 1 backup01 backup01 807 Feb 25 2020 .profile
drwx------ 2 backup01 backup01 4096 Jul 16 11:05 .ssh
-rw-r--r-- 1 backup01 backup01 0 Jul 12 17:12 .sudo_as_admin_successful
-rw------- 1 backup01 backup01 1623 Jul 16 11:02 .viminfo
drwxrwxr-x 6 backup01 backup01 4096 Jul 12 17:43 .xsi
drwxr-xr-x 4 root root 42 Jul 12 17:22 backups

You may see that some folder belongs to the root user. That is because you might have had to create the folder using sudo, which will change ownership to root.

To restore permissions back to the backup01 user, just run this command below.

backup01@LBAK01:~$ sudo chown -R backup01:backup01 /home/backup01

(*) Please, note that you need sudo to change permissions back to backup01:backup01

Running a ©XSIBackup-App backup job from the command line

Now that we have prepared our backup host, exchanged the public key and revised permissions we can run our backup job.

For our test case we will backup VM WIN7-01 sitting at ©ESXi host w.x.y.z. We will need to add the w.x.y.z ©ESXi host to the pool of backup hosts by running the --add-host action. It's pretty simple:

Adding the backup host

[root@XSIBackup-App ~]# xsibackup --add-host root@w.x.y.z:22
System Information: Linux, Kernel 3 Major 10 Minor 0 Patch 1160
Adding host 'w.x.y.z' to the backup pool
Do you want to proceed? (y/n): y
Key exchange routine
This is the routine to exchange keys with other servers
You will be prompted to enter the remote server's password
a number of times. Please do so, this is a one time operation
The remote server is of type: vmkernel OK
Let's look for the 'authorized_keys' file there...
Found 'authorized_keys' file at: /etc/ssh/keys-root/authorized_keys OK
Checking if local public key exists at remote file...
Add local public key to root@w.x.y.z:22:/etc/ssh/keys-root/authorized_keys?
Local public key has been
added to root@w.x.y.z:22:/etc/ssh/keys-root/authorized_keys OK
Remote server successfully linked for user root
Host 'w.x.y.z' was connected and added to the /etc/fstab file

(*) Note how in this case we use the root user to link to the ©ESXi; host. You could create a root close user if you will, but you will need full permissions on your VM folders, especially if you want to write to their configuration, like when using the --enable-cbt action.

Running the backup job

This are the details of the ©ESXi server hosting ©XSIBackup-App and the ©ESXi server hosting the test VM: WIN7-01:

ESXi 5.5 system hosting XSIBackup-App ©ESXi 5.5 system hosting ©XSIBackup-App

As you can see the server above is a 6th generation i5, pretty outdated hardware. We tend to use second hand hardware, not only because it's cheaper, as we use tenths of servers for our tests, but also because it forces us to optimize things even more.

ESXi 6.7 system hosting WIN7-01 ©ESXi 6.7 system hosting WIN7-01

The hypervisor hosting the test VM is even older, a 3rd generation Intel i7, we even had to enable legacy mode flag to install it.

Asumming you can achieve some 20-25 ms latency with your backup host, you will be able to get up to 10-12 MB/s in real sustained throughput. If on addition your virtual machine is not full, but has some empry areas, you will achieve much faster average thoughputs, as the zeros will be skipped.

From the second pass and on, your backup speeds will be much faster, close to the ones you would be getting in a LAN, as only the changed blocks will be sent. If on top of that you use CBT, the effective backup speed will be much higher.

Below two consecutive backups to our Azure Ubuntu backup host in Central France from Spain, that is some 20-22 ms in latency. We changed the IPs by a.b.c.d and w.x.y.z for the remote Azure Linux host and the backup host (the one containing the VM). The rest of the figures are real.