You are not logged in.
You bought XSIBACKUP-PRO, XSIDiff is a binary component of XSIBACKUP-PRO, using XSIDiff as a spare binary is a bonus feature which is not listed among XSIBACKUP-PRO features.
Nevertheless we have checked XSIDiff the way you are using it with big files and it works as expected. Please, debug your network and find why your sockets are reporting errors, the cause is not the software.
We have tried your command in the very same scenario with big files: 50GB and 100GB and we don't get any write error. Debug your network and make sure that it is working properly. XSIDiff will throw those errors when it can't write data to the socket for some reason, the most common one is some network problem.
XSIDiff is not sold alone, it is a binary bundled with XSIBACKUP-PRO, it works when called from XSIBACKUP-PRO. You didn't buy XSIDiff as we don't sell it. Use XSIBACKUP-PRO as described in the program's documentation.
Would you please post the URL where we advertise that XSIBACKUP-PRO will work as a raw data TCP server?
We never announced such feature, in fact the TCP client/server functionality is only announced in the XSIDiff binary command line help, and that's just a bonus feature, which BTW you could have only seen after purchase.
No, Rsync is NOT loaded as client/server, it is indeed tunneled through SSH.
We have already answered why we aren't offering that option.
I would not ignore it for obvious reasons.
XSIDiff, despite its name, is not a differential tool, it just copies data as is. It is zero aware, but it is only differential when used from within Onediff.
When you use it you are copying all data each time. If you get a broken pipe condition in the middle of a copy, restart the job.
A change in the output of some shell commands prevents XSITools pruning from working properly.
This has been fixed in XSIBackup-Pro 11.2.7
That seems to be a simple broken pipe
Well, we have designed the software, you have posed the facts, we have offered you what we believe to be a quite accurate answer. If you don't agree with it we don't know what else to do, as we are leaving the land of facts and entering the land of believing.
XSIBackup main file is just a script that makes use of binaries, so it can hardly make a "bad implementation", but you can easily check that, as it's a script and all logic is there for you to inspect. Nonetheless the best way to compare is to get rid of the script shell and use the XSIDiff binary alone, you will get the same results if you use it with raw data and then over an SSH tunnel.
You are using an Atom CPU, the fact that they named it Avoton won't make it faster.
We have not implemented raw data transfer into XSIBackup cause there exist security standards nowadays. Loading a TCP server would allow anybody to inject data into your ESXi server. We would not even need to have programmed XSIDiff to achieve that, nc is enough.
You do have differential features available and built into XSIBackup that will allow you to overcome the CPU clog, do use them.
O.K., so it was the sunday = 0 that was causing the issue apparently.
ESXi components are like Forrest's box of chocolate, you never know what you're going to get.
We contacted you from the support department to send you an update. Please reach us back to have your issue fixed.
O.K., we saw where the problem is. It's a bug due to changes in the order of the fields of the ls command. We'll correct this today and publish an update as 11.2.7, it will be available tomorrow the latest.
What ESXi version are you using?
It looks like XSIBackup is not able to determine the size of your blocks.
Please run this code against any deduplicated block in your XSITools repository:
ls -1las "/vmfs/volumes/zfs_vmware_backups_dataset/xsibackup/data/0/a/c/0ac859d6f5638..." 2>/dev/null | awk '{print $4}'You must substitute the made up hash (0/a/c/0ac859d6f5638..) by some real hash in your XSITools repo
Remember that you can always split the schedule in multiple lines.
That is an Atom CPU with an extremely limited single core performance. It may scale well for some type of applications, but will never perform well on encryption related tasks. You can consider yourself fortunate that you reach 22 MB/s
Acording to cpubenchmark.com
[url=https://www.cpubenchmark.net/cpu.php?cpu=Intel+Atom+C2750+%40+2.40GHz&id=2185]Intel Atom C2750[/url]
If you compare the single core performance to a plain double core Pentium from 2012 you will see where the problem is
[url=https://www.cpubenchmark.net/cpu.php?cpu=Intel+Pentium+G645+%40+2.90GHz&id=1797]Intel Pemtium G645[/url]
We are sorry to state this, but you have a phisical limiting factor
ESXI cron is not equivalent to a Linux crond, it's a more primitive/basic kind of service, it lacks even an init control script to prevent duplicate execution and other such problems. As VMWare modifies Busybox source code, you can't even be sure of what is current ESXi version supported syntax and you can't trust Busybox documentation either, should it exist.
(c)VMWare does not seem to have published any post regarding the use of the crond service, so we can only guess and use a trial/error approach. The minute, hour and DoM seem to work as expected though.
I'm afraid it is not uderstanding the day range with the hyphen in between (1-6). Try to use commas to separate days. Some older busybox implementations took day abbreviations like: sun, mon, tue...
According to this openwrt.org document making reference to Busybox cron, sunday is 0 and using 7 will lead to commands being run every day.
[url=https://openwrt.org/docs/guide-user/base-system/cron]OpenWRT cron service user guide[/url]
-----------------------------------------------------------------
[b]Tip:[/b]
If you find that single days of the week work as expected, split your command into multiple lines, dirty but effective.
This is the exact piece of code that raises your error at line 1366:
REMESXiVers="$( ( eval ssh "$SSHOPTS" -p "$baksrvport" ${defremusr}@"$baksrvaddr" "\"vmware -v | awk '{print \\\$3}'\"" ) 2>&1 )"You can translate that to
ssh -o PreferredAuthentications=publickey -o StrictHostKeyChecking=no -i xsibackup_id_rsa -p 22 <user>@<ip backup> "vmware -v | awk '{print \$3}'"Run it from your xsi-dir folder, in case you get the permission error, debug by adding -vvv after the ssh binary call.
In case you don't get a permission error, you will need to debug that line in the xsibackup file.
[b]vmware -v[/b] should have execute permissions for your user
Well, we are still lacking some relevant info. What is the remote system you are backing up to?, is it Linux or ESXi.
If the remote System is ESXi you will need to configure a user that has all the appropiate rights on every of the files/folders implied in the backup process at the target, that is: read/write/create permissions on the target folder, which is a full privileged user on the target.
Same will happen on the target system if it's a Linux server.
The way to debug this is to run an SSH command that creates a file at your backup target folder and writes something to it. Enable ssh debugging and inspect the ssh client output.
ssh -vvv -p 22 -o PreferredAuthentications=publickey -o StrictHostKeyChecking=no -i xsibackup_id_rsa <user_backup>@<ip backup> "echo 'HI THERE' > /path/to/backup/directory/test_file.txt"In any case if you performed this test with the root user and it succeded, then it means that your RSA public key was added to the root user [b]authorized_keys[/b] file, and not to the backup user one. Each user has a separate [b]authorized_keys[/b] file where public keys are stored for that user.
Raw data transfer avoiding SSH ciphers is undeniably faster, but the figures you are posting are way too low for XSIDiff over SSH. The only possible explanation is that your CPU is too slow and clogs under the SSH cipher weigth.
Use [b]esxtop[/b] to monitor your CPU while the transfer is taking place.
There is currently no way to use XSIDiff raw transfer from within the XSIBackup user interface, but you may try [b][url=https://33hops.com/xsibackup-vmware-esxi-backup.html]XSIBackup-Datacenter[/url][/b], which is a complete rethink of XSIBackup concept. It has been simpified and ported to native C language to achieve maximum speed.
The [b]./xsibackup --replica[/b] action in [b][url=https://33hops.com/xsibackup-vmware-esxi-backup.html]XSIBackup-Datacenter[/url][/b] is a truly differential replica algorithm, it works both locally and over IP/SSH
O.K., sorry about the "too quick" answer.
The initial premise is wrong, SSH won't be faster than NFS, unless you are connecting to a distant NFS server with high latency and you have configured NFS as syncronous.
You have no other choice than inspecting the authorized_keys file for that user manually, do so and reach as back.
Your XSIBACKUP public RSA key should appear at the bottom
Please read the manual, this forum can't act as a substitute for that.
You have a single line warning that is letting you know exactly what the issue is
Permission denied. Please use --link-srv=remote-IP:remote-port argument to exchange keys.Run the [b][https://33hops.com/xsibackup-help-man-page.html#linksrv]--link-srv[/url][/b] argument to make communication between both servers possible.
You pointed out one of the keystones of chained backups. XSIBackup real power starts to unleash when you take advantage of the many different ways it can be used.
The two key facts to keep in mind are:
- Onediff will use a management snapshot to keep differential data. It will just be kept there in between backup cycles.
- All other backup methods will respect any pre-existing snapshots, they will take their own and backup whatever is there at the time to perform the backup. So the Onediff snapshot will still be there at the time the subsequent Onediff cycle takes place. In other words, you can do whatever in between Onediff cycles, just as long as you don't delete the delta snapshot.
From the two main facts above you can derive your conclusion: yes, you will be backing up the delta snapshot when you perform an XSITools backup and thus you will be keeping it as is. Deduplication will not affect the Onediff delta snapshots, so you will be worsening your compression ratio. Nonetheless, there's one advantage to it, as you will be keeping two restore points per XSITools backup, as you can use the snapshot or discard it.
[b](c)XSIBackup Datacenter[/b] is our next generation solution, it employs the same general principles as XSIBackup-Pro, but it's much simpler to use and it deduplicates everything, including small files.
[url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart](c)XSIBackup-DC product page[/url]
Synology and QNap are both excelent NAS devices manufacturers, but you cannot use their OSs as if they were a regular Linux distros. Root access is limited, even for some basic administration tasks, thus if you try to use them as an OS you will most of the times end up banging your head agains the wall.
You can easily gain access to any storage volume by just using a bridge Linux VM. Attach whatever volume you want to this Linux VM and backup to it over IP. You can attach disks via NFS, iSCSI, SSHFS, etc..., you just need the connected volumes to be available over SSH.
If you try to reach the Synology over SSH you will be hitting the fact that you don't have full root access, you will have to configure additional users and eventually fall into a permissions nightmare. Our recommendation is that you use the synology as an iSCSI or NFS volume and mount those volumes directly to the ESXi server or to some bridge Linux box.
2019-07-14T08:54:05| [winserver] error DIFCRES1: create snapshot xsibackupdiff (1) failed
2019-07-14T08:54:05| [winserver] workaround: could not create quiesced snapshot, trying non-quiesced
2019-07-14T08:54:06| [winserver] warning DIFCRES3: non-quiesced snapshot taken as a workaround measureWell, the message states that XSIBackup could not take a quiesced snapshot, so it tried to take a non-quiesced snapshot and succedded to do so. So the snapshot was indeed created as non quiesced, though.
The commented post explains what you need to know: [url]https://33hops.com/troubleshooting-windows-snapshots-in-esxi.html[/url]
- You first must find out whether you need quiesced snapshots or not. That will depend on the type of services you are running in your Windows server. Tipically database servers will need quiescing.
- Once you know which services will need quiescing, i.e. some SQL Server instance, read about all the services that you need to install to quiesce it in a VMWare ESXi environment and how they must be configured.
- If after installing your software as suggested by the manufacturer, namely MS, you still receive errors at the quiescing stage, look in the event viewer of the Windows guest for errors recorded at the time the quiescing process was happening. The will throw hints about which particular service or driver did not respond in time to the quiescing request. This might indicate that you need to update your VMWare Tools, tweak some time out or that you are still lacking some something in your Windows server.
Onediff is not a backup method, it is actually a replica method, it just keeps a mirrrored VM. Thus if you need multiple restore points, you should perform a backup on addition to Onediff.
The best part is that as Onediff allows you to keep the VM registered on a remote end, performing a backup on this idle VM will not affect performance in your production environment.
So run a Onediff cycle and then run an XSITools backup on the mirrrored VM, that will allow you to keep a virtually unlimited number of restore points on addition to the ready to use Onediff mirror.
[b](c)XSIBackup-DC[/b] is a silent backup service in the background, it does not want all cores in your server, it does not need them, it must not use them, as parallelizing data reads or writes is useless. Your processing power must remain available for your services, not for some service that is overutilizing CPU in your server when it doesn't need it.
[b](c)XSIBackup-DC[/b] does not need more than a pentium single core to achieve maximum speed. You should only bee concerned about your CPU if you are, per instance, using some Atom CPU or some ancient XEON.
It's not all about frequency, it's the CPU architecture that determines how fast a single core will perform, one single core in your server should be enough to reach XSIBackup-Datacenter full potential.
If you find it's not achieving as it should publish here, anyway, first of all make sure that you have activated caché in your controller's BIOS, its such a common issue. HP controllers come with the cache off by default.