#1 2019-07-15 13:11:28

tom_22
Member
Registered: 2019-07-15
Posts: 18

using XSIDiff in client/server mode as XSIBackup backup-prog

Hi,

Is there a way to use XSIDiff in client/server mode as backu-prog for XSIBackup?

If I use XSIDiff in client/server mode, it's very fast. It seems the network link (1Gb) is the limiting device. A 40GB vmdk file takes 207 seconds (197.87 mb/s) to be copied between 2 ESXi Servers.

However, using XSIDiff as backup-prog for XSIBackup, it appears slow. The same 40GB vmdk file takes 1664 seconds (24.62 mb/s), even with the suggested ":l" option applied. I assume the ssh-overhead takes its toll.

Does anyone know how to run XSIDiff in client/server mode for XSIBackup or how to speed up things?

Thanks,
Tom

Offline

#2 2019-07-15 18:52:33

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

Raw data transfer avoiding SSH ciphers is undeniably faster, but the figures you are posting are way too low for XSIDiff over SSH. The only possible explanation is that your CPU is too slow and clogs under the SSH cipher weigth.

Use esxtop to monitor your CPU while the transfer is taking place.

There is currently no way to use XSIDiff raw transfer from within the XSIBackup user interface, but you may try XSIBackup-Datacenter, which is a complete rethink of XSIBackup concept. It has been simpified and ported to native C language to achieve maximum speed.

The ./xsibackup --replica action in XSIBackup-Datacenter is a truly differential replica algorithm, it works both locally and over IP/SSH

Offline

#3 2019-07-16 13:19:25

tom_22
Member
Registered: 2019-07-15
Posts: 18

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

I have used esxtop on both hosts.

Generally, the source host is a C2750 Avoton (8-core), the target host is a C3558 Denverton (4-core).

With just XSIDiff executing in client/server mode, the CPU load average on the source is about 0.14. There are as well 4 VMs running (IDLE). So, I would say hardly no load. This wasn't an issue anyway.

With 'XSIBackup -backup-prog XSIDiff' the CPU load average is about 0.20, the PCPU AVG between 25% and 35%. I've observed a short peak of 68% of one single core, which is probably related to some VM activity.
The target host has a CPU load average of 0.27 and a PCPU AVG of about 30%

The SSH Process takes around 80% (%USED) on the source and around 50% (%USED) on the target. Not sure if this yet is a sign of overload?

I'll have a look at XSIBackup-Datacenter, but would like to get XSIBackup & XSIDiff well performing, as I just bought the license...

Offline

#4 2019-07-16 15:31:33

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

That is an Atom CPU with an extremely limited single core performance. It may scale well for some type of applications, but will never perform well on encryption related tasks. You can consider yourself fortunate that you reach 22 MB/s

Acording to cpubenchmark.com
https://www.cpubenchmark.net/cpu.php?cp … Hz&id=2185

If you compare the single core performance to a plain double core Pentium from 2012 you will see where the problem is
https://www.cpubenchmark.net/cpu.php?cp … Hz&id=1797

We are sorry to state this, but you have a phisical limiting factor

Offline

#5 2019-07-16 22:12:24

tom_22
Member
Registered: 2019-07-15
Posts: 18

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

Well, I don't agree.

XSIDiff works in client/server mode without performance issues. The CPUs almost IDLE.
It's just the implementation into XSIBackup, which is not appropriate. But even then, the CPU/Process values don't peak. The process is around 80% (%USED), but could go well beyond 100% because of dynamic CPU clocking. So one CPU-core is under noticeable workload, but not max'ed out.

There is no indication [https://33hops.com/xsibackup-using-backup-programs.html], that XSIDiff, when used with XSIBackup, is run in a way, so it parse the results from stdin/stdout through ssh & encryption to the other host. How can it then seriously "overtake the initial transfers made by means of Rsync so far"?
Actually, I just tried '--backup-prog=rsync' - it's even faster (31.10MB/s).

Could you please have another look into XSIBackup, so it'll make full use of the advertised features of XSIDiff as a 'backup-prog' option.
I bought the license a few days ago and currently feel a bit disappointed.

Offline

#6 2019-07-17 10:38:33

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

Well, we have designed the software, you have posed the facts, we have offered you what we believe to be a quite accurate answer. If you don't agree with it we don't know what else to do, as we are leaving the land of facts and entering the land of believing.

XSIBackup main file is just a script that makes use of binaries, so it can hardly make a "bad implementation", but you can easily check that, as it's a script and all logic is there for you to inspect. Nonetheless the best way to compare is to get rid of the script shell and use the XSIDiff binary alone, you will get the same results if you use it with raw data and then over an SSH tunnel.

You are using an Atom CPU, the fact that they named it Avoton won't make it faster.

We have not implemented raw data transfer into XSIBackup cause there exist security standards nowadays. Loading a TCP server would allow anybody to inject data into your ESXi server. We would not even need to have programmed XSIDiff to achieve that, nc is enough.

You do have differential features available and built into XSIBackup that will allow you to overcome the CPU clog, do use them.

Offline

#7 2019-07-17 12:49:49

tom_22
Member
Registered: 2019-07-15
Posts: 18

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

Agree, we should stay with the facts: if the CPU is the bottleneck, the numbers should be different.
Please be aware as well, that I haven't written "bad implementation" nor did I expressed this in any other way.

It's advertised, that XSDiff is outperforming rsync. I can reproduce this, if XSDiff is run in client/server mode. The CPUs stay almost in IDLE.
However, XSDiff is slow -even slower than rsync- when used as backup-prog option in XSIBackup, as it is run through a SSH tunnel.

There is a better approach/algorithm in order to achieve, what is advertised. You actually already do this with rsync.

The relevant lines in the script -please correct if wrong- might be this

[XSDIFF]
RSYNCOUT=$( ("$PWD/bin/xsidiff" --source="${flatorigin}" --target=stdout "$XSIDIFFSILENT" | eval ssh "$SSHOPTS" -p$baksrvport ${defremusr}@$baksrvaddr "\"${RMXSIDFPATH//\ /\\ }\" --source=stdin --target=\"${flattarget//\ /\\ }\"" >&16) 2>&1 )

and this

[RSYNC]
RSYNCOUT=$( ("$PWD/bin/xsibackup-rsync" -rlpDv --progress ${rsyncoptions} --whole-file --sparse --rsh="ssh $SSHOPTS -p$baksrvport" --rsync-path="${RMRSYNCPATH}" "$flatorigin" ${defremusr}@$baksrvaddr:"${flattarget}" >&16) 2>&1 )

The rsync command line gives a blueprint (--rsh (-e) option ) of the steps of how to run the XSIDiff command in the very fast client/server mode:

// 1) fire up XSIDiff as server on remote host
ssh name@host "./xsidiff --source=server_IP:33 --target=[file2]"

// 2) run XSIDiff to do the copy/transfer
./xsidiff --source=[file1] --target=server_IP:33

// 3) kill XSIDiff server on remote host
ssh name@host "kill $xsidiff_PID"

What is your thought?

Offline

#8 2019-07-18 11:48:29

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

We have already answered why we aren't offering that option.

Offline

#9 2019-07-18 13:11:20

tom_22
Member
Registered: 2019-07-15
Posts: 18

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

Well, you haven't provided a solution or support on how to make it run as advertised.

You mentioned security standards, that's why you don't load a TCP server.
On the other hand, rsync is loaded that way as client/server. This seems contradictory to us.

We still can't get the tools working (please see other post) as advertised. How to conclude? Should we return the license?

Offline

#10 2019-07-18 15:25:21

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

Would you please post the URL where we advertise that XSIBACKUP-PRO will work as a raw data TCP server?

We never announced such feature, in fact the TCP client/server functionality is only announced in the XSIDiff binary command line help, and that's just a bonus feature, which BTW you could have only seen after purchase.

No, Rsync is NOT loaded as client/server, it is indeed tunneled through SSH.

Offline

#11 2019-07-18 16:56:03

tom_22
Member
Registered: 2019-07-15
Posts: 18

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

https://33hops.com/xsidiff-man-page.html

(c) XSIDiff Man Page:
...
--source
Sets the file to sync/ copy, stdin is allowed. --source can be a local path, stdin or an IP:port combination in case of acting as an IP server.
--target
Sets the destination where the data is to be sent. --target can be a local path, stdout or an IP:port when acting as an IP client.
...

USAGE:
...
IP Server :
./xsidiff --source=10.0.0.1:33 --target=[file2]
IP Client :
./xsidiff --source=[file1] --target=10.0.0.1:33

EXAMPLES:
...
Set server to listen at local IP 192.168.0.100 port 33, copy receiving file to [file2] Once the server is listening run the client setting [file1] as source and point it to the server

./xsidiff --source=192.168.0.100:33 --target=[file2]

Run client when server is listening
./xsidiff --source=[file1] --target=10.0.0.1:33

My take on this situation: we shouldn't quarrel, we should find a solution.

Last edited by tom_22 (2019-07-18 17:16:13)

Offline

#12 2019-07-19 19:20:00

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

You bought XSIBACKUP-PRO, XSIDiff is a binary component of XSIBACKUP-PRO, using XSIDiff as a spare binary is a bonus feature which is not listed among XSIBACKUP-PRO features.

Nevertheless we have checked XSIDiff the way you are using it with big files and it works as expected. Please, debug your network and find why your sockets are reporting errors, the cause is not the software.

Offline

#13 2019-07-28 15:11:27

tom_22
Member
Registered: 2019-07-15
Posts: 18

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

https://33hops.com/xsidiff-vmware-esxi-file-replication.html wrote:

© XSIDiff is as fast as vmkfstools and can copy files over IP.
...
All registered XSIBACKUP-PRO users can request a free license of © XSIDiff to be used as a separate binary tool to copy -flat.vmdk files manually. From the day of launch of XSIBACKUP-PRO 10 (sched. aug 2017), © XSIDiff will be used as a core binary, and may be used separately too. A binary license.key file is necessary to operate with files bigger than 10 gb.

As posted in the other thread: the network looks fine. Copied 120GB file through terminal (cp) to NFS share, from multiple host. We are using Meraki, CISCO and HP-E (as backup) devices. The Switches don't report any lost packet or other issues. No other issues with the network whatsoever. No users reported issues with the network. All other applications work fine.
What other things could be verified in order to see, if there is an issue with the network?

Generally ,'Segmentation fault' is a memory 'access violation' caused/raised by XSDiff.
'Error writing struct info to SOCKET' is related to the socket, which was created by XSIDiff.

We cannot debug the socket, as it is created by XSDiff, but it looks like an overflow.

Last edited by tom_22 (2019-07-28 15:11:51)

Offline

#14 2019-07-29 08:22:30

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

Are you by any chance using NFS4?, NFS4 does not work in the ESXi shell context.
Apart from that try to reboot your servers.

Offline

#15 2019-07-29 14:52:55

tom_22
Member
Registered: 2019-07-15
Posts: 18

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

The NFS Servers run with V3. V4 is disabled.

The (edit: 'relevant') files are all stored on the local datastore, so they are not on mounted (NFS) folders.
The one in question is 128GB large. We have tried another file with 100GB, but unfortunately, when XSDiff is launched on that file, it also throws an 'Segmentation fault' error. (for the sake of completeness: the file with 40GB doesn't result in 'Segmentation fault' )

All vmware servers were rebooted as well.
Also checked the permissions on the files.
Tried on different versions of VMware. Which version are you using?

Last edited by tom_22 (2019-07-29 17:09:23)

Offline

#16 2019-07-30 18:32:50

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

Maybe you are assuming the size has something to do with the issue when it might not. We suspect the metadata layout might be broken or extremely highly fragmented. Although even if every 1M block was fragmente still the metadata layout would fit in the metadata buffer, there's plenty of room there.

We have tested XSIDiff with files way over your size without issue, we just could not reproduce your problem.

Try to clone your disks using Vmkfstools to rebuild the metadata layout that should help the problem in case the issue is related to that.

Offline

#17 2019-08-05 12:42:01

tom_22
Member
Registered: 2019-07-15
Posts: 18

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

We have cloned the disk with vmkfstools and tried again. Unfortunately, still getting the "segmentation fault" error.

Anything else we could do/try?
What VMWare version are you using?

Offline

#18 2019-08-05 18:33:21

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

Please, contact support to solve your particular issue, it probably has to do with your metadata information. I don't think the forum is the best place to post files with thousands of lines.

Execute this code snippet and send us the output as an attachment:

vmkfstools -t0 /path/to/your/problematic/disk.vmdk > /tmp/metadata.txt

Offline

#19 2019-08-07 08:26:59

tom_22
Member
Registered: 2019-07-15
Posts: 18

Re: using XSIDiff in client/server mode as XSIBackup backup-prog

ok. I sent it to "info@33hops.com" and "support@33hops.com".
Please let me know if there is another dedicated address.

Offline

Board footer