You are not logged in.
Hello,
I'm trying to implement XSIBackup DC on one older host with ESXi 5.5, and it's throwing this error on some opertions:
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.5.0.11: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 5 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
License: 000306A90000000000000000e06995d797e0 | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
Remote system: ESXi
-------------------------------------------------------------------------------------------------------------
PID: 522583668, Running job as: root
-------------------------------------------------------------------------------------------------------------
Remote xsibackup binary found at: /scratch/XSI/XSIBackup-DC/xsibackup
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/Backups/backups_ctec
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/Backups/backups_ctec'
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:39:36 | Error code 1500 at file common.c, line 1500 | Error description: could not get remote FS available space at /vmfs/volumes/Backups/backups_ctec, details: No such file or directory
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: -1 (17179.87 PB)
-------------------------------------------------------------------------------------------------------------
pfsense Hardware Version is: 8
-------------------------------------------------------------------------------------------------------------
All snapshots were removed, as pfsense is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:39:40 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) pfsense_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: pfsense
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : pfsense (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Remote target dir is empty, full sync will be performed
-------------------------------------------------------------------------------------------------------------
Virtual Machine: pfsense
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-07-23T15:40:10
-------------------------------------------------------------------------------------------------------------
2021-07-23 15:40:10 | Backing up 21 files, total size is 9.12 GB
-------------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-------------------------------------------------------------------------------------------------------------
1/21 pfsense.vmx 3.17 KB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
2/21 pfsense.vmxf 262.00 B | Done 0.00%
-------------------------------------------------------------------------------------------------------------
3/21 pfsense.vmsd 439.00 B | Done 0.00%
-------------------------------------------------------------------------------------------------------------
4/21 pfsense-flat.vmdk (CBT 1 full) 8.00 GB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 8192 out of 8192 | Done 87.68%
-------------------------------------------------------------------------------------------------------------
5/21 pfsense.vmdk 527.00 B | Done 87.68%
-------------------------------------------------------------------------------------------------------------
6/21 pfsense.nvram 8.48 KB | Done 87.68%
-------------------------------------------------------------------------------------------------------------
7/21 vmware-1.log 129.13 KB | Done 87.68%
-------------------------------------------------------------------------------------------------------------
8/21 vmware-2.log 130.81 KB | Done 87.69%
-------------------------------------------------------------------------------------------------------------
9/21 vmware-3.log 130.72 KB | Done 87.69%
-------------------------------------------------------------------------------------------------------------
10/21 vmx-pfsense-2229485668-1.vswp [open excluded]
-------------------------------------------------------------------------------------------------------------
11/21 pfsense.vmx.lck [skipped excluded]
-------------------------------------------------------------------------------------------------------------
12/21 pfsense.vmx~ 3.16 KB | Done 87.69%
-------------------------------------------------------------------------------------------------------------
13/21 vmware.log 102.71 KB | Done 87.69%
-------------------------------------------------------------------------------------------------------------
14/21 pfsense-84e34064.vswp [open excluded]
-------------------------------------------------------------------------------------------------------------
15/21 pfsense-ctk.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
16/21 pfsense.vmsd.tmp 0.00 B | Done 87.69%
-------------------------------------------------------------------------------------------------------------
17/21 pfsense.vmx.tmp 3.16 KB | Done 87.69%
-------------------------------------------------------------------------------------------------------------
18/21 pfsense-Snapshot1.vmsn 28.08 KB | Done 87.69%
-------------------------------------------------------------------------------------------------------------
19/21 pfsense-000001-delta.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
20/21 pfsense-000001.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
21/21 pfsense-000001-ctk.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
Total size: 8.00 GB | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:13 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:13 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:14 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:15 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:18 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-23T15:46:06
-------------------------------------------------------------------------------------------------------------
Time taken: 00:05:56 (356 sec.)
-------------------------------------------------------------------------------------------------------------
Total time: 356 sec.
-------------------------------------------------------------------------------------------------------------
Full file speed: 23.01 mb/s
-------------------------------------------------------------------------------------------------------------
Real data speed: 67.04 mb/s
-------------------------------------------------------------------------------------------------------------
Item backup completed with errors
-------------------------------------------------------------------------------------------------------------
Item number 2 in this job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:39 | Error code 1500 at file common.c, line 1500 | Error description: could not get remote FS available space at /vmfs/volumes/Backups/backups_ctec, details: No such file or directory
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: -1 (17179.87 PB)
-------------------------------------------------------------------------------------------------------------
Lecturas Hardware Version is: 8
-------------------------------------------------------------------------------------------------------------
All snapshots were removed, as Lecturas is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:43 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) Lecturas_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Lecturas
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : Lecturas (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Remote target dir is empty, full sync will be performed
-------------------------------------------------------------------------------------------------------------
Virtual Machine: Lecturas
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-07-23T15:57:16
-------------------------------------------------------------------------------------------------------------
2021-07-23 15:57:16 | Backing up 22 files, total size is 204.14 GB
-------------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-------------------------------------------------------------------------------------------------------------
1/22 Lecturas.vmx 3.19 KB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
2/22 Lecturas.vmxf 3.19 KB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
3/22 Lecturas.vmsd 442.00 B | Done 0.00%
-------------------------------------------------------------------------------------------------------------
4/22 Lecturas-flat.vmdk (CBT 1 full) 200.00 GB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 204800 out of 204800 | Done 97.97%
-------------------------------------------------------------------------------------------------------------
5/22 Lecturas.vmdk 557.00 B | Done 97.97%
-------------------------------------------------------------------------------------------------------------
6/22 vmware-1.log 590.96 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
7/22 vmware-3.log 165.28 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
8/22 Lecturas.nvram 8.48 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
9/22 vmware-2.log 173.00 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
10/22 vmware-4.log 3.47 MB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
11/22 vmx-Lecturas-1993332819-1.vswp [open excluded]
-------------------------------------------------------------------------------------------------------------
12/22 Lecturas.vmx.lck [skipped excluded]
-------------------------------------------------------------------------------------------------------------
13/22 Lecturas.vmx~ 3.18 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
14/22 vmware.log 197.17 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
15/22 Lecturas-76cfd853.vswp [open excluded]
-------------------------------------------------------------------------------------------------------------
16/22 Lecturas-ctk.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
17/22 Lecturas.vmsd.tmp 0.00 B | Done 97.97%
-------------------------------------------------------------------------------------------------------------
18/22 Lecturas.vmx.tmp 3.18 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
19/22 Lecturas-Snapshot1.vmsn 31.03 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
20/22 Lecturas-000001-delta.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
21/22 Lecturas-000001.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
22/22 Lecturas-000001-ctk.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
Total size: 200.00 GB | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:26 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:26 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-23T18:27:18
-------------------------------------------------------------------------------------------------------------
Time taken: 02:30:02 (9002 sec.)
-------------------------------------------------------------------------------------------------------------
Total time: 9358 sec.
-------------------------------------------------------------------------------------------------------------
Full file speed: 22.75 mb/s
-------------------------------------------------------------------------------------------------------------
Real data speed: 10.13 mb/s
-------------------------------------------------------------------------------------------------------------
Item backup completed with errors
-------------------------------------------------------------------------------------------------------------
Item number 3 in this job
-------------------------------------------------------------------------------------------------------------
-- lines deleted due to forum constraints -
-------------------------------------------------------------------------------------------------------------
2021-07-23T19:54:18 | Error code 4574 at file xsibackup.c, line 4574 | Error description: some error/s were raised while backing up: VMs(pfsense,Lecturas,mysql server,ownCloud), error count is: 28
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
Unlocked backup OK
-------------------------------------------------------------------------------------------------------------
The backup target is a ESXi host version 6.5.0, don't know if that matters.
I tried without CBT and errors persist.
Any suggestion?
Offline
Thank you for your feedback.
Always use the latest release in a branch, in your case ESXi550-201809001 or [b]ESXi U3k[/b].
There have been multiple issues related to shell commands in the past.
Just avoid rotating by size until this is fixed.
We'll revise this to see if we can do something to work the problem around.
Offline
Thanks for your quick answer!
I was trying to have a replica before upgrading :-) I'll go with scp, then upgrade, then XSIBackup doing our regular replicas off-site.
Offline
Did '/vmfs/volumes/Backups/backups_ctec' exist in the remote FS?
It all seems related to an inexistent remote dir, although it should have been created previously.
You may encounter problems when trying to create an SSH tunnel between too distant versions of OpenSSH, as most if not all KEX algorithms and ciphers may have been deprecated.
In this case it looks like they were able to agree on some minimum common denominator in terms of KEX & ciphers, as the remote binary could be updated.
Offline
Yeah, it exists in the remote FS. Indeed, XSIBackup did create it on first backup test. Also the backup somehow works, it seems to fail when removing the CTK info or renaming some files. This is the content of one VM backup dir:
[root@REDACTED:/vmfs/volumes/5ba26f3e-372b4cf8-8bb0-ac1f6b16e426/backups_ctec/Lecturas] ls -la
total 75598464
drwxr-xr-x 1 root root 77824 Jul 24 09:03 .
drwxr-xr-x 1 root root 73728 Jul 23 19:54 ..
drwxr-xr-x 1 root root 77824 Jul 24 09:03 .map
-rw-r--r-- 1 root root 17190912 Jul 24 09:01 Lecturas-000002-delta.vmdk
-rw-r--r-- 1 root root 387 Jul 24 09:01 Lecturas-000002.vmdk
-rw-r--r-- 1 root root 31777 Jul 23 18:38 Lecturas-Snapshot1.vmsn
-rw-r--r-- 1 root root 31777 Jul 24 07:28 Lecturas-Snapshot2.vmsn
-rw-r--r-- 1 root root 31784 Jul 24 09:02 Lecturas-Snapshot3.vmsn
-rw-r--r-- 1 root root 214748364800 Jul 24 08:58 Lecturas-flat.vmdk
-rw-r--r-- 1 root root 8684 Jul 24 09:01 Lecturas.nvram
-rw-r--r-- 1 root root 557 Jul 24 09:00 Lecturas.vmdk
-rw-r--r-- 1 root root 483 Jul 24 08:47 Lecturas.vmsd
-rw-r--r-- 1 root root 77 Jul 24 09:01 Lecturas.vmsd.tmp
-rw-r--r-- 1 root root 3262 Jul 24 08:47 Lecturas.vmx
-rw-r--r-- 1 root root 3262 Jul 24 09:01 Lecturas.vmx.tmp
-rw-r--r-- 1 root root 3263 Jul 24 08:47 Lecturas.vmxf
-rw-r--r-- 1 root root 3262 Jul 24 09:01 Lecturas.vmx~
-rw-r--r-- 1 root root 605141 Jul 24 09:01 vmware-1.log
-rw-r--r-- 1 root root 177153 Jul 24 09:01 vmware-2.log
-rw-r--r-- 1 root root 169246 Jul 24 09:01 vmware-3.log
-rw-r--r-- 1 root root 3634137 Jul 24 09:01 vmware-4.log
-rw-r--r-- 1 root root 443310 Jul 24 09:01 vmware.log
Offline
We aren't quite getting you.
Why do you expect the CBT backup to work when you eliminate the .ctk files?
Why do you expect a --replica to continue to work when you rename files manually?
It's like removing the wheels of your car and then complaining because it doesn't move.
Asuming the rename was done right (that's a different story) you must completely reinitialize the replica.
To reset the CTK files you have the --reset-cbt argument.
Offline
I'm sorry you didn't understood me (maybe my English isn't so good).
I didn't rename or remove anything, I'm talking about the messages logged on the backup:
...
All snapshots were removed, as Lecturas is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:43 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) Lecturas_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Lecturas
....
Total size: 200.00 GB | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:26 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:26 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-23T18:27:18
So, to synthetize:
- I'm trying to backup a ESXi 5.5 host
- XSIBackup tries to remove a VM that didn't exist (Lecturas_XSIREP). I never ran the backup with the argument --options=R
- XSIBackup tries to remove CTK Info from .vmx.tmp and from .vmdk file descriptors / descriptor hashes, and it fails due to a syntax error
- XSIBackup tries to rename .vmx.tmp file to .vmx and .vmsd.tmp file to .vmsd, that fails due to a syntax error
I'm going to try an scp copy and then upgrade the host, but it felt that you should know about these, so that's why I did the report.
Offline
What is the job you are executing?.
You have some kind of bash error. We have tested the software today from ESXi 5.5 to 6.5 with no issue.
Prepend strace to your job to get some detailed system call level output.
strace ./xsibackup ...
Offline
Hello,
This is the command:
strace ./xsibackup --replica=cbt "VMs(Lecturas)" root@REDACTED:22:/vmfs/volumes/Backups/backups_ctec mail-to=REDACTED --subject="Backup ESXi CTEC" --html-template=001 1> /vmfs/volumes/datastore1/xsibackup_strace.log 2>&1
I'm unable to send the full strace log, I tried to remove some parts to 340kb but the forum throws an 404 error.
You can download it from:
[url=https://cloud.mainsip.com/index.php/s/5e7fe5tvdQqRjJo]https://cloud.mainsip.com/index.php/s/5e7fe5tvdQqRjJo[/url]
If you need the original log, please give me an email address as I don't want to put it on a public forum.
Thanks for your support.
Offline
What is this?: "> 1"
Why is the --mail-to argument missing the first "--"?
You are most probably generating your own errors by using a malformed job syntax.
In fact we believe it's quite remarkable that the job took place in the end.
Please, run this and post the output
./xsibackup --replica=cbt "VMs(Lecturas)" root@REDACTED:22:/vmfs/volumes/Backups/backups_ctec
Offline
Hello,
">1" is to send all strace output to a file.
"--mail-to" is missing the "--" due to a copy/paste error.
This is the output of the command:
# ./xsibackup --replica=cbt "VMs(Lecturas)" root@REDACTED:22:/vmfs/volumes/Backups/backups_ctec
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.5.0.11: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 5 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
License: 000306A90000000000000000e06995d797e0 | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
Remote system: ESXi
-------------------------------------------------------------------------------------------------------------
PID: 524375708, Running job as: root
-------------------------------------------------------------------------------------------------------------
Remote xsibackup binary found at: /scratch/XSI/XSIBackup-DC/xsibackup
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/Backups/backups_ctec
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/Backups/backups_ctec'
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:01:55 | Error code 1500 at file common.c, line 1500 | Error description: could not get remote FS available space at /vmfs/volumes/Backups/backups_ctec, details: No such file or directory
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: -1 (17179.87 PB)
-------------------------------------------------------------------------------------------------------------
Lecturas Hardware Version is: 8
-------------------------------------------------------------------------------------------------------------
All snapshots were removed, as Lecturas is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:01:57 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) Lecturas_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Lecturas
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : Lecturas (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: Lecturas
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-07-31T10:02:10
-------------------------------------------------------------------------------------------------------------
2021-07-31 10:02:10 | Backing up 23 files, total size is 204.12 GB
-------------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-------------------------------------------------------------------------------------------------------------
1/23 Lecturas.vmx 3.19 KB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
2/23 Lecturas.vmxf 3.19 KB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
3/23 Lecturas.vmsd 446.00 B | Done 0.00%
-------------------------------------------------------------------------------------------------------------
4/23 Lecturas-flat.vmdk (CBT 147) 200.00 GB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 204800 out of 204800 | Done 97.98%
-------------------------------------------------------------------------------------------------------------
5/23 Lecturas.vmdk 557.00 B | Done 97.98%
-------------------------------------------------------------------------------------------------------------
6/23 vmware-1.log 590.96 KB | Done 97.98%
-------------------------------------------------------------------------------------------------------------
7/23 vmware-3.log 165.28 KB | Done 97.98%
-------------------------------------------------------------------------------------------------------------
8/23 Lecturas.nvram 8.48 KB | Done 97.98%
-------------------------------------------------------------------------------------------------------------
9/23 vmware-2.log 173.00 KB | Done 97.98%
-------------------------------------------------------------------------------------------------------------
10/23 vmware-4.log 3.47 MB | Done 97.98%
-------------------------------------------------------------------------------------------------------------
11/23 vmx-Lecturas-1993332819-1.vswp [open excluded]
-------------------------------------------------------------------------------------------------------------
12/23 Lecturas.vmx.lck [skipped excluded]
-------------------------------------------------------------------------------------------------------------
13/23 Lecturas.vmx~ 3.18 KB | Done 97.98%
-------------------------------------------------------------------------------------------------------------
14/23 vmware.log 1.05 MB | Done 97.98%
-------------------------------------------------------------------------------------------------------------
15/23 Lecturas-76cfd853.vswp [open excluded]
-------------------------------------------------------------------------------------------------------------
16/23 Lecturas-ctk.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
17/23 Lecturas.vmsd.tmp 44.00 B | Done 97.98%
-------------------------------------------------------------------------------------------------------------
18/23 Lecturas.vmx.tmp 3.18 KB | Done 97.98%
-------------------------------------------------------------------------------------------------------------
19/23 Lecturas-aux.xml 13.00 B | Done 97.98%
-------------------------------------------------------------------------------------------------------------
20/23 Lecturas-Snapshot13.vmsn 31.03 KB | Done 97.98%
-------------------------------------------------------------------------------------------------------------
21/23 Lecturas-000001-delta.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
22/23 Lecturas-000001.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
23/23 Lecturas-000001-ctk.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
Total size: 200.01 GB | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:47 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:47 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:47 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:47 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:48 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-31T10:08:51
-------------------------------------------------------------------------------------------------------------
Time taken: 00:06:41 (401 sec.)
-------------------------------------------------------------------------------------------------------------
Total time: 401 sec.
-------------------------------------------------------------------------------------------------------------
Full file speed: 510.74 mb/s
-------------------------------------------------------------------------------------------------------------
Real data speed: 11.18 mb/s
-------------------------------------------------------------------------------------------------------------
Item backup completed with errors
-------------------------------------------------------------------------------------------------------------
Final checksum: 2188883130 bytes were sent and confirmed to have been written remotely
-------------------------------------------------------------------------------------------------------------
2021-07-31T10:09:49 | Error code 4574 at file xsibackup.c, line 4574 | Error description: some error/s were raised while backing up: VMs(Lecturas), error count is: 7
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
Unlocked backup OK
-------------------------------------------------------------------------------------------------------------
I also removed the destination folder to be sure there's nothing in it causing the error. This is the result after removing the folder:
# ./xsibackup --replica=cbt "VMs(Lecturas)" root@REDACTED:22:/vmfs/volumes/Backups/backups_ctec
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.5.0.11: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 5 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
License: 000306A90000000000000000e06995d797e0 | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
Remote system: ESXi
-------------------------------------------------------------------------------------------------------------
PID: 524377608, Running job as: root
-------------------------------------------------------------------------------------------------------------
Remote xsibackup binary found at: /scratch/XSI/XSIBackup-DC/xsibackup
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/Backups/backups_ctec
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/Backups/backups_ctec'
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:11:24 | Error code 1500 at file common.c, line 1500 | Error description: could not get remote FS available space at /vmfs/volumes/Backups/backups_ctec, details: No such file or directory
-------------------------------------------------------------------------------------------------------------
Available space in backup volume: -1 (17179.87 PB)
-------------------------------------------------------------------------------------------------------------
Lecturas Hardware Version is: 8
-------------------------------------------------------------------------------------------------------------
All snapshots were removed, as Lecturas is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:11:26 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) Lecturas_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Lecturas
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : Lecturas (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Remote target dir is empty, full sync will be performed
-------------------------------------------------------------------------------------------------------------
Virtual Machine: Lecturas
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-07-31T10:11:40
-------------------------------------------------------------------------------------------------------------
2021-07-31 10:11:40 | Backing up 23 files, total size is 204.14 GB
-------------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-------------------------------------------------------------------------------------------------------------
1/23 Lecturas.vmx 3.19 KB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
2/23 Lecturas.vmxf 3.19 KB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
3/23 Lecturas.vmsd 447.00 B | Done 0.00%
-------------------------------------------------------------------------------------------------------------
4/23 Lecturas-flat.vmdk (CBT 1 full) 200.00 GB | Done 0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 204800 out of 204800 | Done 97.97%
-------------------------------------------------------------------------------------------------------------
5/23 Lecturas.vmdk 557.00 B | Done 97.97%
-------------------------------------------------------------------------------------------------------------
6/23 vmware-1.log 590.96 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
7/23 vmware-3.log 165.28 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
8/23 Lecturas.nvram 8.48 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
9/23 vmware-2.log 173.00 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
10/23 vmware-4.log 3.47 MB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
11/23 vmx-Lecturas-1993332819-1.vswp [open excluded]
-------------------------------------------------------------------------------------------------------------
12/23 Lecturas.vmx.lck [skipped excluded]
-------------------------------------------------------------------------------------------------------------
13/23 Lecturas.vmx~ 3.18 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
14/23 vmware.log 1.08 MB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
15/23 Lecturas-76cfd853.vswp [open excluded]
-------------------------------------------------------------------------------------------------------------
16/23 Lecturas-ctk.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
17/23 Lecturas.vmsd.tmp 44.00 B | Done 97.97%
-------------------------------------------------------------------------------------------------------------
18/23 Lecturas.vmx.tmp 3.18 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
19/23 Lecturas-aux.xml 13.00 B | Done 97.97%
-------------------------------------------------------------------------------------------------------------
20/23 Lecturas-Snapshot14.vmsn 31.03 KB | Done 97.97%
-------------------------------------------------------------------------------------------------------------
21/23 Lecturas-000001-delta.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
22/23 Lecturas-000001.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
23/23 Lecturas-000001-ctk.vmdk [skipped excluded]
-------------------------------------------------------------------------------------------------------------
Total size: 200.01 GB | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:23 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:23 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:24 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:24 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:24 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-31T12:35:34
-------------------------------------------------------------------------------------------------------------
Time taken: 02:23:54 (8634 sec.)
-------------------------------------------------------------------------------------------------------------
Total time: 8634 sec.
-------------------------------------------------------------------------------------------------------------
Full file speed: 23.72 mb/s
-------------------------------------------------------------------------------------------------------------
Real data speed: 9.67 mb/s
-------------------------------------------------------------------------------------------------------------
Item backup completed with errors
-------------------------------------------------------------------------------------------------------------
Final checksum: 77400130812 bytes were sent and confirmed to have been written remotely
-------------------------------------------------------------------------------------------------------------
2021-07-31T12:36:25 | Error code 4574 at file xsibackup.c, line 4574 | Error description: some error/s were raised while backing up: VMs(Lecturas), error count is: 7
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir OK
-------------------------------------------------------------------------------------------------------------
Unlocked backup OK
-------------------------------------------------------------------------------------------------------------
Offline
It is clear that you can't execute shell commands in the remote system. Make sure that you haven't activated secure boot and that the shell is working, you may need to reinstall ESXi.
There have been shell bugs in the past, try to upgrade your remote ESXi system to the latest patch in the branch.
Offline
Hello,
First, I wanna thank you for your kind support. Much appreciated!
FYI, the remote host is hosting backups from serveral systems correctly, all from XSIBackup, so I think the remote host isn't involved in this problem. All the other hosts doing backups to this same remote host are EXSi 6.5 or higher, that's why I though the ESXi version of the system was involved on the error.
Offline
It could be, we have nonetheless reproduced your environment 5.5 to 6.5 without issue.
Post the output of
vmware -v
From both hosts and we will try to reproduce the exact same environment accurate to the build number.
In any case the problem is rather clear. From your strace.
pipe([10, 11]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x32000910) = 524212574
close(11) = 0
fstat(10, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3ffc0ec7000
read(10, sh: syntax error: unterminated quoted string
"", 600)
The pipe command returns a shell syntax error that isn't happening in other builds nor in our tests. Everything points at some bug in the OpenSSH client.
This is the exact translated command that you can use to debug your scenario and the one that returns the first error. Run from your ESXi 5.5 host and post the output.
ssh -vvv -i xsibackup_id_rsa root@REDACTED "\"/scratch/XSI/XSIBackup-DC/xsibackup\" --free-space \"/vmfs/volumes/Backups/backups_ctec\""
(*) The -vvv will just make ssh client to be verbose.
The problem seems to be related to the double quote escaping, most probably the ssh client is not parsing them well, which would lead us to our very first answer: [b]patch your ESXi 5.5 host[/b].
If you want to make a quick test, just copy some ssh client binary from some close version, maybe the one packed in ESXi 6.0 should work. Still that would not survive a reboot, which would again lead you to the need to patch your host to the latest build in the 5.5 branch.
Offline
We have detected this issue in some other builds, it has to do with the same we comment above. We'll try to work the issue around by running shell commands in some alternative way.
UPDATE:
We can confirm this is due to a bug in the OpenSSL implementation shipped with some versions of (c)ESXi.
The bug consists in that the message "WARNING: can't open config file: /usr/ssl/openssl.cnf" is returned every time you invoke the openssl binary, it has caused some other issues in the past.
We'll work this issue around and publish as 1.5.0.12
Offline
The problem is solved in 1.5.0.12 available in the download zone of the user area.
Please confirm whether this fixed your issue.
Offline