You are not logged in.
so comparing strace between the two systems was leading me to the wrong path regarding xsilib :-(
The sort binary in Synology lacks the T option.
The sort binary in Synology has the T option, but the temp path is missing as argument:
execve("/bin/sh", ["sh", "-c", "sort -T'' | uniq > ../DC-old/data/.blocklog.tmp"], [/* 20 vars */] <unfinished ...>
I removed a lot .map files and I am now down to a volume that --repair is running on VMWare.
It is now creating a blocklog file with 3.853.161 entries.
Giving a max blocksize of 1M that would be not even 4 TB !!
after installing on RockyLinux
[root@rocky klaus]# yum install glibc-devel
[root@rocky klaus]# yum install glibc-devel.i686
I can execute xsilib:
[root@rocky xsi_dir]# bin/xsilib --get-files
Speicherzugriffsfehler (Speicherabzug geschrieben)
but it crashes
some more finding:
I mounted the nfs volume on a Ubuntu 18.04LTS server, getting the same error for the missing temp dir for sort:
sort: option requires an argument -- 'T'
then I installed a brand new RockyLinux system, mounting the nfs volume and starting xsibackup --repair
xsibackup is running and traversing the data dirs, but the first step creating the .blocklog.tmp was skipped: .blocklog.tmp is empty.
Looking at strace I'v found that you are calling xsilib from the bin dir, but:
root@rocky:/mnt/xsibackup01/xsi_dir/bin# ./xsilib
bash: ./xsilib: Datei oder Verzeichnis nicht gefunden
root@rocky:/mnt/xsibackup01/xsi_dir/bin# file ./xsilib
./xsilib: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.9, not stripped
xsilib is a 32bit binary and does not run on 64bit Linux :-(
Thank you for your detailed explanation. Sorry that I didn't explaned our environment in detail in the post.
- Yes, we are backing up to a special Linux based Backup Host (Synology NAS 1817+ with 8x8 TB Disks) mounted over NFS
- The backup volume is 20TB , the occupied space is currently around 12 TB in 16.616.908 files
- I have tried to run --repair from the Linux host:
root@NasBackup01:/volume1/xsibackup01/xsi_dir# ./xsibackup --repair ../DC-old/
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-Free 1.5.1.12: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: Linux, Kernel 3 Major 10 Minor 105 Patch 0
-------------------------------------------------------------------------------------------------------------
PID: 5604, Running job as: root
-------------------------------------------------------------------------------------------------------------
Sorting blocks...
-------------------------------------------------------------------------------------------------------------
sort: option requires an argument -- 'T'
Try 'sort --help' for more information.
execve("/bin/sh", ["sh", "-c", "sort -T'' | uniq > ../DC-old/data/.blocklog.tmp"], [/* 20 vars */] <unfinished ...>
I am trying to repair my repository (about 20TB) but it fails:
[root@Server1:/vmfs/volumes/cacffd7b-11f7dacc/xsi_dir] ./xsibackup --repair /vmfs/volumes/xsibackup01/DC-old/
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.5.1.12: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 7 Major 0 Minor 3 Patch 0
-------------------------------------------------------------------------------------------------------------
PID: 11381218, Running job as: root
-------------------------------------------------------------------------------------------------------------
Sorting blocks...
-------------------------------------------------------------------------------------------------------------
Sorting progress 22%sort: out of memory
-------------------------------------------------------------------------------------------------------------
SIGTERM (13) condition was trapped: check logs for more details
-------------------------------------------------------------------------------------------------------------
Cleaning up...
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
[root@Server1:/vmfs/volumes/cacffd7b-11f7dacc/xsi_dir]
- Hardware : HP DL360 Gen10, 64 GB Memory
- currently only running one VM
- free memory 44 GB
- Sort process allocates about 600m and the crashes
updated to 1.5.1.12 and still having problems creating or editing jobs.
e.g.
- start xsibackup-gui
- Jobs -> Add
- Id : 030
- Action: Backup
- Source: Custom: Select VM
- Target: Local: Select NFS mounted directory
- add description
- add mail-to
- add rotate: 20
- add subfolder
- add use-smtp
- add verbosity
- Save job
Two files were created in ./etc/jobs :
[root@Server1:/tmp/_osdatadqht2t7g/XSI/XSIBackup-DC/etc/jobs] ls -al
-rw-r--r-- 1 root root 21 Apr 7 17:29 030
-rwx------ 1 root root 311 Apr 7 17:31 USV-HQ-Rack1
[root@Server1:/tmp/_osdatadqht2t7g/XSI/XSIBackup-DC/etc/jobs] cat 030
/xsi-dir/xsibackup \
[root@Server1:/tmp/_osdatadqht2t7g/XSI/XSIBackup-DC/etc/jobs] cat USV-HQ-Rack1
/xsi-dir/xsibackup \
--subject="USV-HQ-Rack1" \
--verbosity="6" \
--backup \
"VMs(USV-HQ-Rack1)" \
"/vmfs/volumes/xsibackup01/DC" \
--description="USV-HQ-Rack1" \
--mail-to="administrator@xxxxxxxxxx.de" \
--rotate="20" \
--subfolder="USV-HQ-Rack1" \
--use-smtp="1" \
>> /xsi-dir/var/log/xsibackup.log 2>&1
The hard limit is 998 chars https://datatracker.ietf.org/doc/html/r … tion-2.1.1 but yes it recommends to have only 78 chars.
Having these job variables would be a great thing :-)
my feature request:
Please include the following information in the default email subject:
- Host Name
- Backup Job ID
- VMs / ALL
Yes, I can set a specific subject with the --subject option, but then I will loose the information if there was an error or not.
Best regards
Klaus
PS: The Email text includes the "Session ID" twice but not the job id
I'm doing replica snapshots with DC 1.5.1.5
In the backup folder I'm seeing a lot of files with e.g.
- .lck-dee50200000000
- Hostname.company.local-SnapshotXX.vmsn
(1x vmns per Backup and 1..3 lck per Backup)
who generates these files?
can these files be deleted ?
Best regards
Klaus
Using XSIBackup-DC 1.5.1.3 I'm still having several folders in the tmp dir.
Some of them contain a .blocklog file, some of them are empty.
Thank you for your fast response !! (didn't expect that you are working on a weekend)
Could you please then use the full path including the subdir in the error message ?
Thanks
Klaus
I'm getting the same error message:
2021-12-18T14:21:50 | Error code 3683 at file xsibackup.c, line 3683 | Error description: failed to lock the repo /vmfs/volumes/66633834-7ea5a8ef/SNAPSHOT
When having a backup job of:
/xsi-dir/xsibackup \
--replica=cbt \
"VMs(SQL2012)" \
"/vmfs/volumes/xsibackup02/SNAPSHOT" \
--description="Replica SQL2012" \
--mail-to="administrator@xxxxxxx.de" \
--use-smtp="1" \
--subject="Replica SQL2012" \
--subfolder="SQL2012" \
>> /xsi-dir/var/log/xsibackup.log 2>&1
where is xsibackup placing the .lock file ?
A: /vmfs/volumes/66633834-7ea5a8ef/SNAPSHOT/
B: /vmfs/volumes/66633834-7ea5a8ef/SNAPSHOT/SQL2012/
Creating a new Backup job with GUI creates a job file:
/xsi-dir/xsibackup \
--backup \
"VMs(NavApp3)" \
"/vmfs/volumes/xsibackup02/DC" \
--description="NavApp3" \
--mail-to="administrator@xxxxxxxxxxx.de" \
--subfolder="NavAPP3" \
--use-smtp="1" \
--subject="NavAPP3" \
ooh, I forgot to add the config-backup option, so adding it with the GUI changes to job file to:
/xsi-dir/xsibackup \
--config-backup \
--backup \
"VMs(NavApp3)" \
"/vmfs/volumes/xsibackup02/DC" \
--description="NavApp3" \
--mail-to="administrator@xxxxxxxxxxx.de" \
--subfolder="NavAPP3" \
--use-smtp="1" \
--subject="NavAPP3" \
>> /xsi-dir/var/log/xsibackup.log 2>&1
and now, trying to edit the job again with the GUI gives errors:
+---------------------------------------------------+
| Action argument not recognized, setting to <none> |
| Source argument not recognized, setting to <none> |
| Target argument not recognized, setting to <none> |
+---------------------------------------------------+
Dear Sirs,
after upgrading to 1.5.1.3 I'm seeing a lot of empty directories in ./tmp (I assume for each PID running a xsibackup job)
Could I delete them ?
Historically (full image) backups where needed to recover from hardware failures like a disk crash or fire in the server room.
But today it has an additional purpose:
- restore the data after an attack from a crypto trojaner (virus which encrypts all your data)
Using the technology devloped by 33hops (deduplicating data with an SHA-1 checksum algorithm) it is possible to detect such virus activity.
How?
Usually only a small portion of data is changed in a virtual machine.
E.g. our users share is 500 GB and only 1-2 GByte are usually changing over a day.
As soon as the virus is encrypting the data you will see a sharp rise in the amount of differential data backuped by XSI in the night.
( and no, you can't detect this from a normal file backup running in the virtual machine, as the virus will not change the size of the files or change the archive bit or the last modification time )
So, all you have to do is to monitor the Diff size of the backup (see above excel sheet) (or the relation from Total to Diff size)
( and a central logging system will help you to do this job .. )
PS: I'm not claiming that this is my idea. Some other vendors of backup solutions are already using KI technology to detect such abnormal behavior and raise alarms)
When talking about a DataCenter we are not talking about a single Server running with two or three VMs.
We are talking about several servers with dozens or maybe hundreds of VMs.
One Problem in a DataCenter is managing all these servers and in case of XSIbackup: managing all these backups.
XSIbackup is sending you an email and you can see in the emails subject if the backup was successful or failed.
But as soon as you need some more statistics (e.g. how many data was backuped last weekend) its becoming difficult.
I started to create an Excel sheet and entered some data:
Date Weekday Time Session ID Server Esxi Version XSI Version License Method VM-Name Folder (Destination) Size [GB] Diff [GB] State Duration Speed [MB/s] Compress Error
10.06.2021 4 22:50:03 1111111 Server4 6.0.0 Update 2 (Build 3620759) DC 1.5.0.3 replica (CBT) WinADS001 /vmfs/volumes/nas1/backup1 380 9,39 1 00:04:47 1355,82 1 0
10.06.2021 4 23:00:34 1111112 Server5 6.0.0 Update 2 (Build 3620759) DC 1.5.0.3 replica (CBT) WinADS002 /vmfs/volumes/nas2/backup2 480 2,98 1 00:00:21 23406,14 1 0
10.06.2021 4 23:02:07 1111113 Server4 6.0.0 Update 2 (Build 3620759) DC 1.5.0.3 replica (CBT) AppServer1 /vmfs/volumes/nas1/backup1 80 79,5 1 00:31:58 42,75 1 0
11.06.2021 5 01:37:07 1111114 Server9 7.0.0 (Build 16324942) DC 1.5.0.3 replica (CBT) AppServer2 /vmfs/volumes/nas1/backup1 40 39,38 1 00:06:59 97,76 1 0
11.06.2021 5 00:01:24 1111115 Server8 6.0.0 Update 2 (Build 3620759) DC 1.5.0.3 backup Docker001 /vmfs/volumes/nas1/backup1 80 4,39 1 00:01:16 1078,13 1 0
But thats a time-consuming work and therefore I would like to raise the following proposal:
- Send the backup infos to an central logging server
A DataCenter has usually one central logging server who collects e.g. syslogs from router and switches
or event logs from windows servers or application logs from web servers.
Some of these tools used are:
- Logstash, Elasticseach and Kibana
- GrayLog
(see https://www.tecmint.com/open-source-cen … nt-tools/)
Integrating XSIBackup in this logging infrastructure is no big deal:
- create a log string
(either using CSV or tab delimited data like the columns in my excel sheet, or a JSON string)
- send it to through VMWare's syslog
- or send it directly to the log server using either TCP or UDP to a specified port
so, how could I send you the log ?
Well, that was very foreseable, now the question is, who or what?
I think you are questioning who or what modified the remote replicated VM?
That's a good question, as the destination is on a Synology NAS Storage and is not mounted to any VMWare host.
I have tried to reset-cbt but seeing the problem again on these VMs:
- Qlik-VM
- Zoll-VM
I may send you the full xsibackup.log (I have found no way to attach a file to this post..)
PS: see also: https://33hops.com/forum/viewtopic.php?id=968
You are right:
/!\ Some changes were detected in the remote replicated VM a full sync will be performed now.
I will try to --reset-cbt and report later
I am sending the backup logs by smtp to me,
but it is cumbersome to have a review how the backup jobs of the last week/month for a special VM were performed.
Would it be possible to have a local log (eg. in xsi-dir/var/log) per backupjob-id or better per job-description?
Example: ./var/log/WinADS2.log
Date Time JobID Action VM-Name Size Diff. State Time Speed Compress Errors
2021-05-29 21:06:12 002 backup [46] WinADS2 480.01 GB 469.72 GB 1 13:05:52 10.42 MB/s 1 0
2021-06-01 23:00:26 001 replica [46] WinADS2 480.01 GB 2.76 MB 1 00:00:15 32768.58 MB/s 1 0
2021-06-02 23:00:21 001 replica [46] WinADS2 480.01 GB 2.78 MB 1 00:00:11 44684.43 MB/s 1 0
as we had a public holiday yesterday I expected that there would only small diffs in the VMs.
But I'm seeing some VMs with large diffs (almost full backup):
XSIBackup-DC 1.5.0.3
Action: replica=cbt
CBT enabled for all VMs
The Good:
UTC | Thu, 03 Jun 2021 23:00:29
1 [46] WinADS2 480.01 GB 2.81 MB 1 00:00:16 30720.55 MB/s 1 0
UTC | Wed, 02 Jun 2021 23:00:21
1 [46] WinADS2 480.01 GB 2.78 MB 1 00:00:11 44684.43 MB/s 1 0
UTC | Tue, 01 Jun 2021 23:00:26
1 [46] WinADS2 480.01 GB 2.76 MB 1 00:00:15 32768.58 MB/s 1 0
The Bad:
UTC | Fri, 04 Jun 2021 00:48:24
1 [77] Zoll-VM 60.01 GB 47.71 GB 1 00:18:16 56.06 MB/s 1 0
UTC | Thu, 03 Jun 2021 00:49:18
1 [77] Zoll-VM 60.01 GB 47.66 GB 1 00:19:10 53.43 MB/s 1 0
UTC | Wed, 02 Jun 2021 00:47:30
1 [77] Zoll-VM 60.01 GB 47.66 GB 1 00:17:23 58.91 MB/s 1 0
Zoll-VM is a Windows 10 VM with only 35% used disk space:
PS C:\Users\Zoll> Get-PSDrive
Name Used (GB) Free (GB) Provider Root
---- --------- --------- -------- ----
C 24,64 34,75 FileSystem C:\
How can I diagnose why there is such a large diff ?
as we only have one SMTP-server and all emails should be send to administrator,
I thought instead of repeating use-smtp/mail-to in every job I could drop this
in the [defaults] section in xsibackup.conf.
But it wasn't used during the backup job, so if this is not a bug,
I would like to post this as a feature request.
Best regards
Klaus
Please don't get me wrong, I love XSIBackup, we are using it for now over 4 years and its fantastic that you created XSIBackup-DC.
But I am just curious
1. how good will XSIBackup-DC perform
2. why does the backup of our SQL-Server machine takes so long
I am doing performance analysis for now over 30 years, so I am fully aware of the points you mentioned above.
So my setup is quite simple:
- HP DL360 Gen10 with 8 CPUs x Intel(R) Xeon(R) Silver 4110
- 2 x Synology NAS RS820RP+ with 4 x Samsung 1,8 TB SSD RAID
- connected to a single Cisco SG350X switch
> It's not only the theoretical 88MB/s bottleneck
The 88MB/s is not a theoretical figure but was measured on this system running IOPerf with 16Kb blocksize
Running a dd job which copies the data from one storage to the other WITHIN a virtual machine gives even better figures:
root@buddgie-vm:/mnt/source# time dd if=../source/bigfile of=../target/bigfile bs=1M
9216+0 Datensätze ein
9216+0 Datensätze aus
9663676416 Bytes (9,7 GB, 9.0 GiB) kopiert, 92,1166 s, 105 MB/s
real 1m32,121s
user 0m0,046s
sys 0m25,742s
I tried cp and dd on the esxi shell too, and they achived the same speed as xsibackup: around 34 MB/S
There is a old case study from Intel from 2010 (11 years ago!): https://www.intel.com/content/dam/suppo … yfinal.pdf
where they used scp and rsync and stated:
"but the standard tools are not very well threaded, so they don't take full advantage of .. this hardware platform"
and recommend: "Choose the right tools .. and use more parallelism when possible"
( I had the problem on Windows too, when I tried to migrate a mailstore with millions of small files to a new host.
The standard tools were to slow, so I developed a tool which uses multiple threads and large buffers and is able to saturate the 1GB link)
So I am curious if xsibackup uses a multithreaded, buffered architecture
or is there really an artifical performance slowdown for the esxi shell ? ( someone mentioned this on stackexchange)
What are the best figures you are getting with a 1GB link?
( And yes, I am just talking about the speed for the first run of "xsibackup --replica" )
Thank you for your good work!
Klaus
PS: I will try to setup some tests using backup over IP. If you are interested I may report about these tests
Hi rob,
as we are talking about the vmware world: a nfs datastore is mounted to the host system and then it is looking and handled like any other local datastore.
So you could use Vmkfstools, but the best backup program is: XSIBackup !!
Hi rob,
crond is the process who looks into the crontabs and starts the processes according to the defined schedules.
There is only ONE crond process needed to do this work, so everything is ok.
See also: https://linux.die.net/man/8/crond