©XSIBackup-Free: Free Backup Software for ©VMWare ©ESXi

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software


You are not logged in.

#1 Re: General matters » XSIBackup-DC 1.5.1.12: --repair crashes with "sort: out of memory" » 2022-04-09 15:30:09

so comparing strace between the two systems was leading me to the wrong path regarding xsilib :-(

The sort binary in Synology lacks the T option.

The sort binary in Synology has the T option, but the temp path is missing as argument:

execve("/bin/sh", ["sh", "-c", "sort -T'' | uniq > ../DC-old/data/.blocklog.tmp"], [/* 20 vars */] <unfinished ...>

I removed a lot .map files and I am now down to a volume that --repair is running on VMWare.
It is now creating a blocklog file with 3.853.161 entries.
Giving a max blocksize of 1M that would be not even 4 TB !!

#2 Re: General matters » XSIBackup-DC 1.5.1.12: --repair crashes with "sort: out of memory" » 2022-04-08 20:08:15

after installing on RockyLinux

[root@rocky klaus]# yum install glibc-devel
[root@rocky klaus]# yum install glibc-devel.i686

I can execute xsilib:

[root@rocky xsi_dir]# bin/xsilib --get-files
Speicherzugriffsfehler (Speicherabzug geschrieben)

but it crashes

#3 Re: General matters » XSIBackup-DC 1.5.1.12: --repair crashes with "sort: out of memory" » 2022-04-08 18:19:52

some more finding:

I mounted the nfs volume on a Ubuntu 18.04LTS server, getting the same error for the missing temp dir for sort:

sort: option requires an argument -- 'T'

then I installed a brand new RockyLinux system, mounting the nfs volume and starting xsibackup --repair
xsibackup is running and traversing the data dirs, but the first step creating the .blocklog.tmp was skipped: .blocklog.tmp is empty.

Looking at strace I'v found that you are calling xsilib from the bin dir, but:

root@rocky:/mnt/xsibackup01/xsi_dir/bin# ./xsilib
bash: ./xsilib: Datei oder Verzeichnis nicht gefunden
root@rocky:/mnt/xsibackup01/xsi_dir/bin# file ./xsilib
./xsilib: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.9, not stripped

xsilib is a 32bit binary and does not run on 64bit Linux :-(

#4 Re: General matters » XSIBackup-DC 1.5.1.12: --repair crashes with "sort: out of memory" » 2022-04-08 15:18:51

Thank you for your detailed explanation. Sorry that I didn't explaned our environment in detail in the post.

- Yes, we are backing up to a special Linux based Backup Host (Synology NAS 1817+ with 8x8 TB Disks) mounted over NFS
- The backup volume is 20TB , the occupied space is currently around 12 TB in 16.616.908 files
- I have tried to run --repair from the Linux host:

root@NasBackup01:/volume1/xsibackup01/xsi_dir# ./xsibackup --repair ../DC-old/
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-Free 1.5.1.12: Backup & Replication Software                 |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: Linux, Kernel 3 Major 10 Minor 105 Patch 0
-------------------------------------------------------------------------------------------------------------
PID: 5604, Running job as: root
-------------------------------------------------------------------------------------------------------------
Sorting blocks...
-------------------------------------------------------------------------------------------------------------
sort: option requires an argument -- 'T'
Try 'sort --help' for more information.
 execve("/bin/sh", ["sh", "-c", "sort -T'' | uniq > ../DC-old/data/.blocklog.tmp"], [/* 20 vars */] <unfinished ...>

#5 General matters » XSIBackup-DC 1.5.1.12: --repair crashes with "sort: out of memory" » 2022-04-08 09:07:48

kd.gundermann
Replies: 8

I am trying to repair my repository (about 20TB) but it fails:

[root@Server1:/vmfs/volumes/cacffd7b-11f7dacc/xsi_dir] ./xsibackup --repair /vmfs/volumes/xsibackup01/DC-old/
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.5.1.12: Backup & Replication Software                   |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 7 Major 0 Minor 3 Patch 0
-------------------------------------------------------------------------------------------------------------
PID: 11381218, Running job as: root
-------------------------------------------------------------------------------------------------------------
Sorting blocks...
-------------------------------------------------------------------------------------------------------------
Sorting progress 22%sort: out of memory

-------------------------------------------------------------------------------------------------------------
SIGTERM (13) condition was trapped: check logs for more details
-------------------------------------------------------------------------------------------------------------
Cleaning up...
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
[root@Server1:/vmfs/volumes/cacffd7b-11f7dacc/xsi_dir]

- Hardware : HP DL360 Gen10, 64 GB Memory
- currently only running one VM
- free memory 44 GB
- Sort process allocates about 600m and the crashes

#6 Re: General matters » XSIBackup-DC 1.5.1.3 : Error parsing job » 2022-04-07 17:40:01

updated to 1.5.1.12 and still having problems creating or editing jobs.
e.g.
- start xsibackup-gui
- Jobs -> Add
- Id : 030
- Action: Backup
- Source: Custom: Select VM
- Target: Local: Select NFS mounted directory
- add description
- add mail-to
- add rotate: 20
- add subfolder
- add use-smtp
- add verbosity
- Save job

Two files were created in ./etc/jobs :

[root@Server1:/tmp/_osdatadqht2t7g/XSI/XSIBackup-DC/etc/jobs] ls -al
-rw-r--r--    1 root     root            21 Apr  7 17:29 030
-rwx------    1 root     root           311 Apr  7 17:31 USV-HQ-Rack1

[root@Server1:/tmp/_osdatadqht2t7g/XSI/XSIBackup-DC/etc/jobs] cat 030
/xsi-dir/xsibackup \
[root@Server1:/tmp/_osdatadqht2t7g/XSI/XSIBackup-DC/etc/jobs] cat USV-HQ-Rack1
/xsi-dir/xsibackup \
--subject="USV-HQ-Rack1" \
--verbosity="6" \
--backup \
"VMs(USV-HQ-Rack1)" \
"/vmfs/volumes/xsibackup01/DC" \
--description="USV-HQ-Rack1" \
--mail-to="administrator@xxxxxxxxxx.de" \
--rotate="20" \
--subfolder="USV-HQ-Rack1" \
--use-smtp="1" \
>> /xsi-dir/var/log/xsibackup.log 2>&1

#7 Re: Feature requests & improvements » Email subject » 2022-04-06 20:36:07

The hard limit is 998 chars https://datatracker.ietf.org/doc/html/r … tion-2.1.1 but yes it recommends to have only 78 chars.

Having these job variables would be a great thing :-)

#8 Feature requests & improvements » Email subject » 2022-04-06 10:32:00

kd.gundermann
Replies: 3

my feature request:

Please include the following information in the default email subject:
- Host Name
- Backup Job ID
- VMs / ALL

Yes, I can set a specific subject with the --subject option, but then I will loose the information if there was an error or not.

Best regards

Klaus

PS: The Email text includes the "Session ID" twice but not the job id

#9 General matters » Lots of .lck-xxxxxx Files » 2022-02-25 14:21:00

kd.gundermann
Replies: 1

I'm doing replica snapshots with DC 1.5.1.5

In the backup folder I'm seeing a lot of files with e.g.
- .lck-dee50200000000
- Hostname.company.local-SnapshotXX.vmsn

(1x vmns per Backup and 1..3 lck per Backup)

who generates these files?
can these files be deleted ?

Best regards

Klaus

#10 Re: General matters » xsibackup doesn't cleanup tmp folder » 2021-12-18 16:56:19

Using XSIBackup-DC 1.5.1.3 I'm still having several folders in the tmp dir.
Some of them contain a .blocklog file, some of them are empty.

#11 Re: General matters » Can two XSIbackup instances use the same repository simultaneously? » 2021-12-18 16:44:14

Thank you for your fast response !! (didn't expect that you are working on a weekend)

Could you please then use the full path including the subdir in the error message ?

Thanks

Klaus

#12 Re: General matters » Can two XSIbackup instances use the same repository simultaneously? » 2021-12-18 14:28:34

I'm getting the same error message:

2021-12-18T14:21:50 | Error code 3683 at file xsibackup.c, line 3683 | Error description: failed to lock the repo /vmfs/volumes/66633834-7ea5a8ef/SNAPSHOT

When having a backup job of:

/xsi-dir/xsibackup \
--replica=cbt \
"VMs(SQL2012)" \
"/vmfs/volumes/xsibackup02/SNAPSHOT" \
--description="Replica SQL2012" \
--mail-to="administrator@xxxxxxx.de" \
--use-smtp="1" \
--subject="Replica SQL2012" \
--subfolder="SQL2012" \
>> /xsi-dir/var/log/xsibackup.log 2>&1

where is xsibackup placing the .lock file ?

A:  /vmfs/volumes/66633834-7ea5a8ef/SNAPSHOT/
B:  /vmfs/volumes/66633834-7ea5a8ef/SNAPSHOT/SQL2012/

#13 General matters » XSIBackup-DC 1.5.1.3 : Error parsing job » 2021-12-03 12:46:32

kd.gundermann
Replies: 3

Creating a new Backup job with GUI creates a job file:

/xsi-dir/xsibackup \
--backup \
"VMs(NavApp3)" \
"/vmfs/volumes/xsibackup02/DC" \
--description="NavApp3" \
--mail-to="administrator@xxxxxxxxxxx.de" \
--subfolder="NavAPP3" \
--use-smtp="1" \
--subject="NavAPP3" \

ooh, I forgot to add the config-backup option, so adding it with the GUI changes to job file to:

/xsi-dir/xsibackup \
--config-backup \
--backup \
"VMs(NavApp3)" \
"/vmfs/volumes/xsibackup02/DC" \
--description="NavApp3" \
--mail-to="administrator@xxxxxxxxxxx.de" \
--subfolder="NavAPP3" \
--use-smtp="1" \
--subject="NavAPP3" \
>> /xsi-dir/var/log/xsibackup.log 2>&1

and now, trying to edit the job again with the GUI gives errors:

+---------------------------------------------------+
| Action argument not recognized, setting to <none> |
| Source argument not recognized, setting to <none> |
| Target argument not recognized, setting to <none> |
+---------------------------------------------------+

#14 General matters » XSIBackup-DC 1.5.1.3 : empty dirs in xsi-dir/tmp » 2021-11-12 19:20:27

kd.gundermann
Replies: 1

Dear Sirs,

after upgrading to 1.5.1.3 I'm seeing a lot of empty directories in ./tmp (I assume for each PID running a xsibackup job)
Could I delete them ?

#15 Re: Feature requests & improvements » Central Logging » 2021-06-21 20:21:59

Why I am doing backups ?

Historically (full image) backups where needed to recover from hardware failures like a disk crash or fire in the server room.

But today it has an additional purpose:
- restore the data after an attack from a crypto trojaner (virus which encrypts all your data)

Using the technology devloped by 33hops (deduplicating data with an SHA-1 checksum algorithm) it is possible to detect such virus activity.

How?
Usually only a small portion of data is changed in a virtual machine.
E.g. our users share is 500 GB and only 1-2 GByte are usually changing over a day.
As soon as the virus is encrypting the data you will see a sharp rise in the amount of differential data backuped by XSI in the night.
( and no, you can't detect this from a normal file backup running in the virtual machine, as the virus will not change the size of the files or change the archive bit or the last modification time )

So, all you have to do is to monitor the Diff size of the backup (see above excel sheet) (or the relation from Total to Diff size)
( and a central logging system will help you to do this job .. )

PS: I'm not claiming that this is my idea. Some other vendors of backup solutions are already using KI technology to detect such abnormal behavior and raise alarms)

#16 Feature requests & improvements » Central Logging » 2021-06-21 19:59:26

kd.gundermann
Replies: 2

When talking about a DataCenter we are not talking about a single Server running with two or three VMs.
We are talking about several servers with dozens or maybe hundreds of VMs.

One Problem in a DataCenter is managing all these servers and in case of XSIbackup: managing all these backups.

XSIbackup is sending you an email and you can see in the emails subject if the backup was successful or failed.
But as soon as you need some more statistics (e.g. how many data was backuped last weekend) its becoming difficult.

I started to create an Excel sheet and entered some data:

Date	      Weekday	Time	  Session ID	Server	Esxi Version 		        XSI Version	License	Method		VM-Name	Folder (Destination)		Size [GB]	Diff [GB]	State	Duration	Speed [MB/s]	Compress	Error
10.06.2021	4	22:50:03	1111111	Server4	6.0.0 Update 2 (Build 3620759)	DC 1.5.0.3		replica (CBT)	WinADS001	/vmfs/volumes/nas1/backup1	380	9,39		1	00:04:47	1355,82		1	0
10.06.2021	4	23:00:34	1111112	Server5	6.0.0 Update 2 (Build 3620759)	DC 1.5.0.3		replica (CBT)	WinADS002	/vmfs/volumes/nas2/backup2	480	2,98		1	00:00:21	23406,14	 1	0
10.06.2021	4	23:02:07	1111113	Server4	6.0.0 Update 2 (Build 3620759)	DC 1.5.0.3		replica (CBT)	AppServer1	/vmfs/volumes/nas1/backup1	80	79,5		1	00:31:58	42,75		1	0
11.06.2021	5	01:37:07	1111114	Server9	7.0.0 (Build 16324942) 		DC 1.5.0.3		replica (CBT)	AppServer2	/vmfs/volumes/nas1/backup1	40	39,38		1	00:06:59	97,76	1	0
11.06.2021	5	00:01:24	1111115	Server8	6.0.0 Update 2 (Build 3620759)	DC 1.5.0.3		backup      	Docker001	/vmfs/volumes/nas1/backup1	80	4,39		1	00:01:16	1078,13		1	0

But thats a time-consuming work and therefore I would like to raise the following proposal:
- Send the backup infos to an central logging server

A DataCenter has usually one central logging server who collects e.g. syslogs from router and switches
or event logs from windows servers or application logs from web servers.

Some of these tools used are:
- Logstash, Elasticseach and Kibana
- GrayLog
(see https://www.tecmint.com/open-source-cen … nt-tools/)

Integrating XSIBackup in this logging infrastructure is no big deal:
- create a log string
  (either using CSV or tab delimited data like the columns in my excel sheet, or a JSON string)
- send it to through VMWare's syslog
- or send it directly to the log server using either TCP or UDP to a specified port

#18 Re: General matters » Replica with large diff size » 2021-06-15 11:21:43

admin wrote:

Well, that was very foreseable, now the question is, who or what?

I think you are questioning who or what modified the remote replicated VM?
That's a good question, as the destination is on a Synology NAS Storage and is not mounted to any VMWare host.

I have tried to reset-cbt but seeing the problem again on these VMs:
- Qlik-VM
- Zoll-VM

I may send you the full xsibackup.log (I have found no way to attach a file to this post..)

PS: see also: https://33hops.com/forum/viewtopic.php?id=968

#19 Re: General matters » Replica with large diff size » 2021-06-07 09:35:45

You are right:

 /!\ Some changes were detected in the remote replicated VM a full sync will be performed now. 

I will try to --reset-cbt and report later

#20 Feature requests & improvements » Logs per Job » 2021-06-04 14:37:47

kd.gundermann
Replies: 1

I am sending the backup logs by smtp to me,
but it is cumbersome to have a review how the backup jobs of the last week/month for a special VM were performed.

Would it be possible to have a local log (eg. in xsi-dir/var/log) per backupjob-id or better per job-description?

Example:  ./var/log/WinADS2.log

Date       Time      JobID  Action     VM-Name         Size       Diff.        State Time          Speed          Compress  Errors
2021-05-29 21:06:12  002    backup   [46] WinADS2     480.01 GB  469.72 GB       1    13:05:52  10.42 MB/s       1               0
2021-06-01 23:00:26  001    replica  [46] WinADS2     480.01 GB  2.76 MB	 1    00:00:15  32768.58 MB/s    1               0
2021-06-02 23:00:21  001    replica  [46] WinADS2     480.01 GB  2.78 MB	 1    00:00:11  44684.43 MB/s    1               0

#21 General matters » Replica with large diff size » 2021-06-04 14:18:58

kd.gundermann
Replies: 7

as we had a public holiday yesterday I expected that there would only small diffs in the VMs.
But I'm seeing some VMs with large diffs (almost full backup):

XSIBackup-DC 1.5.0.3
Action: replica=cbt
CBT enabled for all VMs

The Good:

UTC | Thu, 03 Jun 2021 23:00:29
1	[46] WinADS2	480.01 GB	2.81 MB	1	00:00:16	30720.55 MB/s	1	0
UTC | Wed, 02 Jun 2021 23:00:21
1	[46] WinADS2	480.01 GB	2.78 MB	1	00:00:11	44684.43 MB/s	1	0
UTC | Tue, 01 Jun 2021 23:00:26
1	[46] WinADS2	480.01 GB	2.76 MB	1	00:00:15	32768.58 MB/s	1	0

The Bad:

UTC | Fri, 04 Jun 2021 00:48:24
1	[77] Zoll-VM	60.01 GB	47.71 GB	1	00:18:16	56.06 MB/s	1	0
UTC | Thu, 03 Jun 2021 00:49:18
1	[77] Zoll-VM	60.01 GB	47.66 GB	1	00:19:10	53.43 MB/s	1	0
UTC | Wed, 02 Jun 2021 00:47:30
1	[77] Zoll-VM	60.01 GB	47.66 GB	1	00:17:23	58.91 MB/s	1	0

Zoll-VM is a Windows 10 VM with only 35% used disk space:

PS C:\Users\Zoll> Get-PSDrive

Name           Used (GB)     Free (GB) Provider      Root                                               
----           ---------     --------- --------      ----
C                  24,64         34,75 FileSystem    C:\

How can I diagnose why there is such a large diff ?

#22 Feature requests & improvements » XSIBackup-DC 1.5 : [defaults] section in xsibackup.conf » 2021-05-27 11:52:04

kd.gundermann
Replies: 1

as we only have one SMTP-server and all emails should be send to administrator,
I thought instead of repeating use-smtp/mail-to in every job I could drop this
in the [defaults] section in xsibackup.conf.

But it wasn't used during the backup job, so if this is not a bug,
I would like to post this as a feature request.

Best regards

Klaus

#23 Re: Speed » XSIBackup-DC Perf Test » 2021-05-26 13:27:52

Please don't get me wrong, I love XSIBackup, we are using it for now over 4 years and its fantastic that you created XSIBackup-DC.

But I am just curious
1. how good will XSIBackup-DC perform
2. why does the backup of our SQL-Server machine takes so long

I am doing performance analysis for now over 30 years, so I am fully aware of the points you mentioned above.
So my setup is quite simple:
- HP DL360 Gen10 with 8 CPUs x Intel(R) Xeon(R) Silver 4110
- 2 x Synology NAS RS820RP+ with 4 x Samsung 1,8 TB SSD RAID
- connected to a single Cisco SG350X switch

> It's not only the theoretical 88MB/s bottleneck

The 88MB/s is not a theoretical figure but was measured on this system running IOPerf with 16Kb blocksize

Running a dd job which copies the data from one storage to the other WITHIN a virtual machine gives even better figures:

root@buddgie-vm:/mnt/source# time dd if=../source/bigfile of=../target/bigfile bs=1M
9216+0 Datensätze ein
9216+0 Datensätze aus
9663676416 Bytes (9,7 GB, 9.0 GiB) kopiert, 92,1166 s, 105 MB/s

real   1m32,121s
user   0m0,046s
sys     0m25,742s

I tried cp and dd on the esxi shell too, and they achived the same speed as xsibackup: around 34 MB/S

There is a old case study from Intel from 2010 (11 years ago!): https://www.intel.com/content/dam/suppo … yfinal.pdf
where they used scp and rsync and stated:
"but the standard tools are not very well threaded, so they don't take full advantage of .. this hardware platform"
and recommend: "Choose the right tools .. and use more parallelism when possible"

( I had the problem on Windows too, when I tried to migrate a mailstore with millions of small files to a new host.
  The standard tools were to slow, so I developed a tool which uses multiple threads and large buffers and is able to saturate the 1GB link)

So I am curious if xsibackup uses a multithreaded, buffered architecture
or is there really an artifical performance slowdown for the esxi shell ? ( someone mentioned this on stackexchange)
What are the best figures you are getting with a 1GB link?

( And yes, I am just talking about the speed for the first run of "xsibackup --replica" )

Thank  you for your good work!

Klaus

PS: I will try to setup some tests using backup over IP. If you are interested I may report about these tests

#24 Re: General matters » best backup program to use for nfs datastores » 2021-05-25 11:59:31

Hi rob,

as we are talking about the vmware world: a nfs datastore is mounted to the host system and then it is looking and handled like any other local datastore.
So you could use Vmkfstools, but the best backup program is: XSIBackup !!

#25 Re: General matters » has it updated crontab? (solved) » 2021-05-25 11:55:01

Hi rob,

crond is the process who looks into the crontabs and starts the processes according to the defined schedules.
There is only ONE crond process needed to do this work, so everything is ok.

See also: https://linux.die.net/man/8/crond

Board footer