Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

kd.gundermann

Member

Registered: 2018-02-27

Posts: 34

Central Logging

When talking about a DataCenter we are not talking about a single Server running with two or three VMs.
We are talking about several servers with dozens or maybe hundreds of VMs.

One Problem in a DataCenter is managing all these servers and in case of XSIbackup: managing all these backups.

XSIbackup is sending you an email and you can see in the emails subject if the backup was successful or failed.
But as soon as you need some more statistics (e.g. how many data was backuped last weekend) its becoming difficult.

I started to create an Excel sheet and entered some data:

Date	      Weekday	Time	  Session ID	Server	Esxi Version 		        XSI Version	License	Method		VM-Name	Folder (Destination)		Size [GB]	Diff [GB]	State	Duration	Speed [MB/s]	Compress	Error
10.06.2021	4	22:50:03	1111111	Server4	6.0.0 Update 2 (Build 3620759)	DC 1.5.0.3		replica (CBT)	WinADS001	/vmfs/volumes/nas1/backup1	380	9,39		1	00:04:47	1355,82		1	0
10.06.2021	4	23:00:34	1111112	Server5	6.0.0 Update 2 (Build 3620759)	DC 1.5.0.3		replica (CBT)	WinADS002	/vmfs/volumes/nas2/backup2	480	2,98		1	00:00:21	23406,14	 1	0
10.06.2021	4	23:02:07	1111113	Server4	6.0.0 Update 2 (Build 3620759)	DC 1.5.0.3		replica (CBT)	AppServer1	/vmfs/volumes/nas1/backup1	80	79,5		1	00:31:58	42,75		1	0
11.06.2021	5	01:37:07	1111114	Server9	7.0.0 (Build 16324942) 		DC 1.5.0.3		replica (CBT)	AppServer2	/vmfs/volumes/nas1/backup1	40	39,38		1	00:06:59	97,76	1	0
11.06.2021	5	00:01:24	1111115	Server8	6.0.0 Update 2 (Build 3620759)	DC 1.5.0.3		backup      	Docker001	/vmfs/volumes/nas1/backup1	80	4,39		1	00:01:16	1078,13		1	0

But thats a time-consuming work and therefore I would like to raise the following proposal:
- Send the backup infos to an central logging server

A DataCenter has usually one central logging server who collects e.g. syslogs from router and switches
or event logs from windows servers or application logs from web servers.

Some of these tools used are:
- Logstash, Elasticseach and Kibana
- GrayLog
(see [url]https://www.tecmint.com/open-source-centralized-linux-log-management-tools/)[/url]

Integrating XSIBackup in this logging infrastructure is no big deal:
- create a log string
  (either using CSV or tab delimited data like the columns in my excel sheet, or a JSON string)
- send it to through VMWare's syslog
- or send it directly to the log server using either TCP or UDP to a specified port

kd.gundermann

Member

Registered: 2018-02-27

Posts: 34

Re: Central Logging

[h]Why I am doing backups ?[/h]

Historically (full image) backups where needed to recover from hardware failures like a disk crash or fire in the server room.

But today it has an additional purpose:
- restore the data after an attack from a crypto trojaner (virus which encrypts all your data)

Using the technology devloped by 33hops (deduplicating data with an SHA-1 checksum algorithm) it is possible to detect such virus activity.

How?
Usually only a small portion of data is changed in a virtual machine.
E.g. our users share is 500 GB and only 1-2 GByte are usually changing over a day.
As soon as the virus is encrypting the data you will see a sharp rise in the amount of differential data backuped by XSI in the night.
( and no, you can't detect this from a normal file backup running in the virtual machine, as the virus will not change the size of the files or change the archive bit or the last modification time )

So, all you have to do is to monitor the Diff size of the backup (see above excel sheet) (or the relation from Total to Diff size)
( and a central logging system will help you to do this job .. )

PS: I'm not claiming that this is my idea. Some other vendors of backup solutions are already using KI technology to detect such abnormal behavior and raise alarms)

admin

Administrator

Registered: 2017-04-21

Posts: 2,072

Re: Central Logging

What you say does make sense. You can actually add the information generated during a backup cycle to let's say some MySQL DB easily, you just have to exchange key with the DB server and run a postbackup script to add this info to the database.

We will dedicate some time to writing some posts and offer some examples that can be reused by (c)XSIBackup users and also some DB triggers to generate alerts on the situation that you describe.

We are actually finishing to test some new features around --rotate to allow setting virtual user space limits and improving the rotation features.