Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

[b](c)XSIBackup-DC 1.4.3.0 RC1[/b] is ready for those of you willing to use it.
It is fully functional during 10 days, so you can try it at any server.

Current (c)XSIBackup-Free available for download is not compatible with (c)ESXi Free.
We will release New (c)XSIBackup-Free, based in DC technology in a couple of days.

Rename your disks and make each one of them unique to the VM, or include part of the path that allows to differentiate them.

There are some thing that we need to fine tune relative to the --add-key argument, next version will include a fix.
It will come out in a few days. To work your issue around:

1 - Use the old version to exchange the keys and then overwrite the xsibackup binary with the latest.
2 - Add the public key (xsibackup_id_rsa.pub) to the remote server's authorized_keys file manually:

cat xsibackup_id_rsa.pub | ssh root@a.b.c.d "cat >> /etc/ssh/keys-root/authorized_keys"

The code snippet above will do it, just substitute the a.b.c.d fake IP address by your remote server's one.

3 - We forgot to mention that the same function in the GUI is working properly, so you just have to use the GUI for adding keys to remote systems, that's a lot easier than the above.

Are you quoting paths with spaces?
It could be something specific to your ESXi build. We have detected different bugs in the command line binaries, in fact some of them related to spaces. An upgrade to the latest build in the branch should be enough.

Now you know why we insist so much in you: sysadmins, taking things easy and renouncing to spaces in paths, long unjustified names, special characters and so on.... If you pledge to that way of doing things you will save time enough to master playing electric guitar, or anything else you like.

(c)XSIBackup-Pro suppports spaces in file names.

Don't look over our previous post, we are pointing at the right way to handle your situation.
When you have some intensive I/O service, it needs to be stopped before the snapshot and started right after. (c)VMWare Tools provide very specific ways to do it manually when you can't rely on quiescing.

Merry Christmas to you too!!

There's always plenty of activity at any given server, but that service does not seem to be something generic, it's probably some other application that makes use of it. You may not even need it to be running. You will have to find that out.

We have already gone too far with this issue. You are being affected by an everyday's IT Admin life general issue which is not directly related to ©XSIBackup, but to quiescing the FS.

If you can't solve this on your own we can help you, we would charge for our time though.

Well, we don't even know what application that chunk of log above corresponds to:
Does [b]SoapAdapter.HTTPService[/b] mean something to you?.
It could be some other error, are you sure it's raised right at the time to take the snapshot?.

All we can see it's an error related to something communicating on a file descriptor that seems to be related to a process communicating on TCP in localhost (127.0.0.1) on ports 8307, 36575

If that is the culprint, then you might be running some service at the same time the snapshot is being taken, which is not compatible with VMWare Tools quiescing.

The solution is simple, keep it iddle during that backup window or use some type of backup that does not require quiescing, like warm or cold backup types.

Please note, that using warm or cold, does indeed quiesce the FS the tough way, that is: stopping the VM. In case of warm backups, a snapshot is taken from off state and then the VM is rapidly switched on again, thus the downtime is minimal.

If your VM is gently switched off, you will have the equivalent of a total service quiescing state (as all services will be orderly turned off), while, if your VM needs to be powered off (due to the same service not responding to a shutdown request), you will have the same situation as if you would plainly issue a non-quiesced snapshot.

Determining whether the abrupt power off or non-quiesced snapshot can compromise the integrity of your data will depend on knowing the "rebel" service in more detail, namely: how intelligent it is at the time to manage that kind of situation.

UPDATE:

We are setting up some lab environment to try to reproduce your issue. This is an update on what we have found so far:

-    The system call fail collected by strace on your ©ESXi host is clear: ENOSPC (No space left on device). That is: we have hit a system limit.
-    Attached you will find a list of ©ESXi 6.5’s error codes. ENOSPC is used in multiple places, but it’s always related to limit of inodes reached or lack of memory to store file descriptors…
[url=https://virtualization24x7.blogspot.com/2017/03/table-of-all-vmkernel-error-codes-in.html]ENOSPC Error[/url]
-    There exist multiple bugs in ©ESXi related to that error. In fact ©ESXi 6.7 was buggy at the beginning, we hit some of those bugs while developing ©XSIBackup-Pro.
-    [url=https://communities.vmware.com/t5/ESXi-Discussions/Vmware-Esxi-Update-No-Space-Left-On-Device-Error/td-p/1409499]No space left on device errors[/url]
-    If we are able to reproduce the issue on multiple ©ESXi versions, that means we have hit some hardcoded limit and that we need to rethink how some parts DC work to allow it to go further, as it did up to 1.4.0.0. In newer versions we improved indexing of the .blocklog file by adding a more complex dynamic indexing system. That makes DC faster, but it is more resource intensive. If on the contrary the issue is only reproduceable in your ©ESXi build, then that would mean that you just hit an ©ESXi bug and an upgrade would solve the issue.

Also:

Recheck that your NFS bond is working properly and that you are not storing your backup repositories in the root FS instead of in the NFS server, just in case.

Simply using a bigger block size, like 10MB should help work the problem around while we study the way to take maximum advantage of the available resources.

The problem is clear, do not use non-ASCII characters in datastore names.

Of course it is important.
In fact it is indeed the first error in a stack that points in the right direction.
We can't obviously help you if you don't post the relevant information. You have now posted the error, but you have skipped the rest of the log, thus we don't know: ESXi version, local and remote FSs, etc...
If you are concerned about posting that data, then contact support, but some comment is not enough to open a serious ticket.

There's something ackward somewhere in your df -h output (used by (c)XSIBackup-DC internally).

  File "/sbin/df", line 110, in <module>
    sys.exit(main(sys.argv))
  File "/sbin/df", line 106, in main
    i["Percentage"], i["Mounted on"])
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 88: ordinal not in range(128)

The above error is thrown by the Python stack at ESXi, which is used for many internal management tasks and is pointing at some encoding issue in the path of some datastore. In fact this error is below what (c)XSIBackup-DC can manage, so don't even try to perform a backup until you get read of that error.

The error is self explanatory, it's even letting you know the character that is causing the issue: [b]u'\xfc'[/b]
If you cut&paste data between user interfaces you can provoke this kind of situation without even knowing.
For some reason that non-ASCII character has got into some datastore name, most probably.

A way to reproduce that error would be to just run the [b]df -h[/b] command from the SSH command line of your (c)ESXi host.