Registered users
Linkedin Twitter Facebook Google+
Close

In order to improve user's experience and to enable some functionalities by tracking the user accross the website, this website uses its own cookies and from third parties, like Google Analytics and other similar activity tracking software. Read the Privacy Policy
33HOPS, IT Consultants
33HOPS ::: Proveedores de Soluciones Informáticas :: Madrid :+34 91 663 6085Avda. Castilla la Mancha, 95 - local posterior - 28700 S.S. de los Reyes - MADRID33HOPS, Sistemas de Informacion y Redes, S.L.Info

<< Return to index

XSIBACKUP-PRO Out of the Box #2 | Basic Backup Job

We should start this series of case studies by analyzing the most frequent type of VM backup. We'll call this a "Basic Backup Job". The case consists on a number of VMs, which I will reduce to three, that will need to be backed up to a local datastore in order to be able to recover them in case of a disaster recovery situation. This is something basic, we would not be covered in case of a fire or a theft, but you could combine it a with a basic procedure by which you use two different portable disks and always keep one out of the office, per instance.

We will be backing up this three VMs (Linux1, Windows1, Unix1) every night. Our office is a regular SME with 33 employees or a bigger enterprise regional branch office, that uses this three virtualized servers in a double CPU Xeon server equipped with three 4 tb. SATA drives plus a 500 mb SSD that will be used as a caché disk. On top of that we have a Synology or QNap NAS server with the same amount of disks. This is a very common hardware configuration in any SME or regional branch around the globe.



Let's make things clear

I'm not the type of guy that writes convenience papers, I don't think you have seen any banner in this site, and I'm not willing to act in any firm's behalf. In fact what I am about to say does not harm anybody. Most of the hardware manufacturers out there create usable and useful pieces of hardware, just as long as you use them for the purpose they were meant for.

I could concrete and say: "don't expect to reach high transfer rates if you use home meant network hardware, it's not XSIBackup's fault if you are backing up at 10 mb/s over your gigabit NICs". You can certainly run ESXi servers on commodity hardware and still reach good average backup speeds, but you should not save in network equipment. That does not mean you have to buy multi thousand network switches or spend hundreds of dollars in your NICs, it just means that the switch and the server's NICs are crucial, and that you need to choose them well.

I highly recommend Intel NICs, almost any server NIC chipset is well made and will be recognized by ESXi (check the ESXi hardware compatibility list before buying). In regards to the switches you need to design a good topology that allows to avoid bottlenecks, a common topology is to use a powerful switch at a first level, to allow the servers to communicate between them and one or more switches to manage the network client's traffic. You can also install one additional double port NIC in the server and connect it directly to the backup NAS device, this last option is damn simple and equally effective.

In regards to NIC teaming, don't get too excited because you have four NICs in your server, teaming might not work the way you expected. It's true that building a NIC team is quite straight most of the times, but there are a number of ways to set up a NIC team, and the bandwidth does not sum up as is. First of all, if you want NIC teaming, your switch needs to support it, so you'll have to read the papers and choose your hardware accordingly to what you pretend to achieve. As an excerpt, NIC bonding will offer you various benefits like failover and load balancing, but that does not mean that when you transfer a huge .vmdk file it will be being transmitted through all your NIC ports at the same time and thus you'll be multiplying your speed by the number of bonded ports.

In respect to the NAS server, don't complicate your life, Synology and QNap devices are well made and optimized for data transfer, it's not worth to build your own NAS on top of a server unless you already have a spare one or have the need of a very powerful NAS in terms of CPU and/or memory, as would be the case if you wanted to run some FS like ZFS.

The backup design

Once we have gone through the most important facts when dealing with the hardware, we can focus in the backup logic. In first place, we should realize that we are destining valuable resources to backup our data. Part of our data will never change, per instance, a bunch of hi resolution photos from our annual meeting in 2007 is probably something we don't want to backup every day. Thus, all obsolete data has to be extracted and archived before even posing any backup strategy. Secondly, a backup is not a backup if it does not allow you to go back in time, at least a number of days, you should also keep at least some weekly backups and also some monthly ones. Apart from that you can keep other backups from previous years offsite just for archival purposes. Copying your server to the same place every night is not an option, as you would replicate any problem to your copy.

What are the most dangerous threats in today's world?:

Obviously classic threats are still there: fire, theft, broken water pipes and so on..., and we also have ever changing threats, like: malware, trojans and specially ransom ware, I know a couple of clients that have been affected by ransom ware incidents in the last year, and the only thing that saved them was having a good set of backups.

- So, the first basic characteristic a backup design must have is: multiple backups available, the more the better.

Apart from that thought, there are other important facts to take on account. Will we need to perform hot backups? (while the VMs are running), or, can we switch the VMs off because there's nobody here during the night?. If the office is not busy during the night making backups will be a kiddies game, so we will turn our case study company into a 33 seat 24x7 Call Center, to make the case study a little bit more interesting. So, yes, we will be needing to perform hot backups, as stated above we have three servers:

- A Windows server: it's a PDC that also has the CEO, CIO and two more executives home folders, so it's acting both as a PDC and file server. It is installed in a single 300 gb .vmdk file and has 200 gb. of data plus 100 gb. of empty space.

- A Linux server: it holds the rest of the staff data, mainly stored in a CRM database of about 50 gb., but there are also file folders belonging to 30 Call Center operators, summing up 300 gb, a HHRR database of about 10 gb. and some DDBB logs and technical data. This server is installed in a 500 gb. HD and holds 370 gb. of data plus 130 gb. of empty space.

- A Unix server: this is the PBX that gives service to the Call Center, it is installed in a FreeBSD OS and holds the PBX plus 100 gb. of recordings from the daily conversations, summing up 110 gb of data plus 290 gb of empty space into a 400 gb. vmdk HD.

O.K., so we have 1.2 tb. of data to backup every day, and we must do it in such way that nobody notices it. We are ready to design our backup strategy with XSIBACKUP-PRO.

THE BACKUP JOBS PROGRAMMING:

Well, first of all do not panic, we used the word programming more in the sense of scheduling, you don't need to be a programmer to use XSIBACKUP-PRO.

Let's start by the Windows server. We know it is the PDC, so we cannot switch it off. Apart from that we know that the executives that use it as a file server are not at the office from 17:00 to 07:30 h.. Copying those 200 gb. while the rest of the stuff is using the Linux Server and the FreeBSD PBX might cause the PBX to drop the calls and the file server to work sluggish. If we use XSIBACKUP-PRO, we will be working at the ESXi shell level, thus, we will be only using one of the available cores in the server, thus the impact in terms of CPU will be very low. The memory management has been optimized to only use very limited amounts of RAM, so the only thing that we should be afraid of is clogging the RAID controller or the network cards with the stream of data, which is not despicable (200 gb).

Do you remember some paragraphs above when I proposed to use a separate NIC to hold the backup traffic?, that is a cheap way to detour backup data traffic from the production NICs, the ones that will be used to keep the office working. If you use that technique, you will be causing a low impact in the CPU and memory and also a low impact in the NIC cards. But wait, what about the HDs attached to the server?, we could be saturating the disk controller channel with our backup traffic. The obvious solution to this is to isolate, as much as possible, the work units in different HDs attached to different disk controller channels.

In our case, the correct connection of the disks should be: one disk holding the PDC and executive folders server .vmdk disk attached to the disk controller's channel number 1. The Linux server should be spread in, at least, two .vmdk disks, one of them holding the database files connected to the disk controller's channel number 2 and the other holding the user files connected to the disk controllers channel number 3. The PBX's .vmdk disks should be connected to the last disk controller's channel, number 3 too. Assuming we have 4 channels in our SATA controller, we can use the fourth to connect the 500 gb. SSD disk, that will help the system run smoother.

This disposition of things will use our hardware resources rationally, and will allow to perform a backup without disrupting the other departments' productivity. There are other ways to backup our servers that will be much less exigent in terms of data transfer, but we will leave that for next posts, by now we will copy all data every time, that will force us to squeeze our brains to optimize the systems layer.

O.K., now that you have reconnected everything to take advantage of your hardware to the maximum extent, let's see how your ESXi server looks from inside. You should have a number of data stores (4), three corresponding to the SATA disks and one corresponding to your NAS device. I recommend that you mount it through NFS, it's simpler and more efficient than iSCSI, at least with the various different NAS devices that we have tested. Make sure that you have access via SSH to your ESXi's server's shell. Login to your server through SSH and run: df -h, you should see something like the following:



I will now start to add the XSIBACKUP-PRO jobs we will need to backup all our VMs to the xsibackup-cron file. We have 12T of backup space and each day we will be backing up 1.2 tb. of data, then we should be able to fit 9 days of backups there. As said, we will start by the Windows PDC. All backup jobs must be tested from the command line before adding them to the crontab.



Backup job explained:

• As the executives leave the office at 17:30 and don't come back until 07:30 am, we have a plenty wide open backup window here, we will start our backup job at 19:00 h. just to make sure that the chances there's somebody still working are smaller.

• As the executives don't normally work during the weekends, we won't be backing up this server on Saturdays and Sundays, as data won't normally grow those days.

• We don't need to parse the --backup-how=hot option, as it is the default behaviour.

• I will be explicitly using the --backup-vms option to set the VMs to backup, as we are backing up just one VM with this backup job (Windows1).

• We well be sending a backup report to administrator@callcenter.com by means of the SMTP server #1 set in the conf/smtpsrvs file.

• The --date-dir=yes argument/option has been set, thus backups will be stored in timestamped folders under the --backup-point directory.

• And finally, we are setting the --on-success and --on-error arguments, so that when the backup job ends, a second backup job (backupid->02) is started.


Now we need to set up the second backup job (backupid=02), that will be fired when the first backup job ends. Taking on account the way work is organized in our Call Center, and that the first backup (200 gb), will last about half an hour to 45 min., we should probably be thinking about launching the Linux server backup, and leave the PBX backup to be made nightly, as the volume of incoming calls in the Call Center normally decreases.



• In this case we won't be adding the --time argument, as the second backup job depends on when the first finishes and is triggered by its end.

• This backup job will use the second server configured in the SMTP servers file, just for fun.

The second backup job (backupid=02) will last about 2-3 h., it doubles the size of the first backup volume and will be made through a pretty busy controller channel, thus we should expect it to be a bit slower. You should also take on account that the stream of backup data will slow down the server response, still it should work fast enough to keep production going.

And last, but not least, we need to run our third backup, which will be a lot similar to the previous, but will not add the --on-success and --on-error arguments. We should expect this third backup to be launched at about 22:00 h. which would be O.K.. Nevertheless you could alternatively remove the --on-success="backupid->03" --on-error="backupid->03" in the second backup and set an execution time for the last backup, setting it at 03:00 h. per instance. The time gap is big enaough to make sure that backups don't overlap. Take on account that XSIBACKUP-PRO does not allow parallel execution of backups. It's not a limitation, we have programmed it like that on purpose. You may wonder why, well, it does not matter how big or powerfull your server is, storage is the slowest part of any server and launching parallel terabyte backups is not recommendable, as it may very well clogg your server.

Daniel J. García Fidalgo
33HOPS



Website Map
Consultancy
IT Manager
In Site
Resources & help
Index of Docs
33HOPS Forum

Fill in to download
The download link will be sent to your e-mail.
Name
Lastname
E-mail


            Read our Privacy Policy