©XSIBackup-Free: Free Backup Software for ©VMWare ©ESXi

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software


You are not logged in.

#1 2021-07-24 08:49:26

underdpt
Member
Registered: 2018-04-30
Posts: 19

Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

Hello,

I'm trying to implement XSIBackup DC on one older host with ESXi 5.5, and it's throwing this error on some opertions:

|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.5.0.11: Backup & Replication Software                   |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 5 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
License: 000306A90000000000000000e06995d797e0 | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
Remote system: ESXi
-------------------------------------------------------------------------------------------------------------
PID: 522583668, Running job as: root
-------------------------------------------------------------------------------------------------------------
Remote xsibackup binary found at: /scratch/XSI/XSIBackup-DC/xsibackup
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/Backups/backups_ctec
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/Backups/backups_ctec'
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:39:36 | Error code 1500 at file common.c, line 1500 | Error description: could not get remote FS available space at /vmfs/volumes/Backups/backups_ctec, details: No such file or directory
-------------------------------------------------------------------------------------------------------------
Available space in backup volume:           -1 (17179.87 PB)
-------------------------------------------------------------------------------------------------------------
pfsense Hardware Version is: 8
-------------------------------------------------------------------------------------------------------------
All snapshots were removed, as pfsense is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:39:40 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) pfsense_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: pfsense
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : pfsense (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Remote target dir is empty, full sync will be performed
-------------------------------------------------------------------------------------------------------------
Virtual Machine: pfsense
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-07-23T15:40:10
-------------------------------------------------------------------------------------------------------------
2021-07-23 15:40:10 | Backing up 21 files, total size is 9.12 GB
-------------------------------------------------------------------------------------------------------------
    NUMBER                                                         FILE             SIZE          PROGRESS
-------------------------------------------------------------------------------------------------------------
    1/21                                                    pfsense.vmx          3.17 KB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    2/21                                                   pfsense.vmxf        262.00 B     | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    3/21                                                   pfsense.vmsd        439.00 B     | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    4/21                                 pfsense-flat.vmdk (CBT 1 full)          8.00 GB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 8192 out of 8192                                        | Done  87.68%
-------------------------------------------------------------------------------------------------------------
    5/21                                                   pfsense.vmdk        527.00 B     | Done  87.68%
-------------------------------------------------------------------------------------------------------------
    6/21                                                  pfsense.nvram          8.48 KB    | Done  87.68%
-------------------------------------------------------------------------------------------------------------
    7/21                                                   vmware-1.log        129.13 KB    | Done  87.68%
-------------------------------------------------------------------------------------------------------------
    8/21                                                   vmware-2.log        130.81 KB    | Done  87.69%
-------------------------------------------------------------------------------------------------------------
    9/21                                                   vmware-3.log        130.72 KB    | Done  87.69%
-------------------------------------------------------------------------------------------------------------
   10/21                                  vmx-pfsense-2229485668-1.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   11/21                                                pfsense.vmx.lck                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   12/21                                                   pfsense.vmx~          3.16 KB    | Done  87.69%
-------------------------------------------------------------------------------------------------------------
   13/21                                                     vmware.log        102.71 KB    | Done  87.69%
-------------------------------------------------------------------------------------------------------------
   14/21                                          pfsense-84e34064.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   15/21                                               pfsense-ctk.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   16/21                                               pfsense.vmsd.tmp          0.00 B     | Done  87.69%
-------------------------------------------------------------------------------------------------------------
   17/21                                                pfsense.vmx.tmp          3.16 KB    | Done  87.69%
-------------------------------------------------------------------------------------------------------------
   18/21                                         pfsense-Snapshot1.vmsn         28.08 KB    | Done  87.69%
-------------------------------------------------------------------------------------------------------------
   19/21                                      pfsense-000001-delta.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   20/21                                            pfsense-000001.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   21/21                                        pfsense-000001-ctk.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
Total size:                                                                      8.00 GB    | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:13 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:13 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:14 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:15 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:18 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-23T15:46:06
-------------------------------------------------------------------------------------------------------------
Time taken: 00:05:56 (356 sec.)
-------------------------------------------------------------------------------------------------------------
Total time:      356 sec.
-------------------------------------------------------------------------------------------------------------
Full file speed:                                                                            23.01 mb/s
-------------------------------------------------------------------------------------------------------------
Real data speed:                                                                            67.04 mb/s
-------------------------------------------------------------------------------------------------------------
Item backup completed with errors
-------------------------------------------------------------------------------------------------------------
Item number 2 in this job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:39 | Error code 1500 at file common.c, line 1500 | Error description: could not get remote FS available space at /vmfs/volumes/Backups/backups_ctec, details: No such file or directory
-------------------------------------------------------------------------------------------------------------
Available space in backup volume:           -1 (17179.87 PB)
-------------------------------------------------------------------------------------------------------------
Lecturas Hardware Version is: 8
-------------------------------------------------------------------------------------------------------------
All snapshots were removed, as Lecturas is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:43 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) Lecturas_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Lecturas
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : Lecturas (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Remote target dir is empty, full sync will be performed
-------------------------------------------------------------------------------------------------------------
Virtual Machine: Lecturas
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-07-23T15:57:16
-------------------------------------------------------------------------------------------------------------
2021-07-23 15:57:16 | Backing up 22 files, total size is 204.14 GB
-------------------------------------------------------------------------------------------------------------
    NUMBER                                                         FILE             SIZE          PROGRESS
-------------------------------------------------------------------------------------------------------------
    1/22                                                   Lecturas.vmx          3.19 KB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    2/22                                                  Lecturas.vmxf          3.19 KB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    3/22                                                  Lecturas.vmsd        442.00 B     | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    4/22                                Lecturas-flat.vmdk (CBT 1 full)        200.00 GB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 204800 out of 204800                                    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    5/22                                                  Lecturas.vmdk        557.00 B     | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    6/22                                                   vmware-1.log        590.96 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    7/22                                                   vmware-3.log        165.28 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    8/22                                                 Lecturas.nvram          8.48 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    9/22                                                   vmware-2.log        173.00 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   10/22                                                   vmware-4.log          3.47 MB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   11/22                                 vmx-Lecturas-1993332819-1.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   12/22                                               Lecturas.vmx.lck                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   13/22                                                  Lecturas.vmx~          3.18 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   14/22                                                     vmware.log        197.17 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   15/22                                         Lecturas-76cfd853.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   16/22                                              Lecturas-ctk.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   17/22                                              Lecturas.vmsd.tmp          0.00 B     | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   18/22                                               Lecturas.vmx.tmp          3.18 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   19/22                                        Lecturas-Snapshot1.vmsn         31.03 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   20/22                                     Lecturas-000001-delta.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   21/22                                           Lecturas-000001.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   22/22                                       Lecturas-000001-ctk.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
Total size:                                                                    200.00 GB    | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:26 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:26 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-23T18:27:18
-------------------------------------------------------------------------------------------------------------
Time taken: 02:30:02 (9002 sec.)
-------------------------------------------------------------------------------------------------------------
Total time:     9358 sec.
-------------------------------------------------------------------------------------------------------------
Full file speed:                                                                            22.75 mb/s
-------------------------------------------------------------------------------------------------------------
Real data speed:                                                                            10.13 mb/s
-------------------------------------------------------------------------------------------------------------
Item backup completed with errors
-------------------------------------------------------------------------------------------------------------
Item number 3 in this job
-------------------------------------------------------------------------------------------------------------

-- lines deleted due to forum constraints -

-------------------------------------------------------------------------------------------------------------

2021-07-23T19:54:18 | Error code 4574 at file xsibackup.c, line 4574 | Error description: some error/s were raised while backing up: VMs(pfsense,Lecturas,mysql server,ownCloud), error count is: 28
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Unlocked backup               OK
-------------------------------------------------------------------------------------------------------------

The backup target is a ESXi host version 6.5.0, don't know if that matters.

I tried without CBT and errors persist.

Any suggestion?

Offline

#2 2021-07-24 09:57:32

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

Thank you for your feedback.

Always use the latest release in a branch, in your case ESXi550-201809001 or ESXi U3k.
There have been multiple issues related to shell commands in the past.
Just avoid rotating by size until this is fixed.

We'll revise this to see if we can do something to work the problem around.

Offline

#3 2021-07-28 06:15:49

underdpt
Member
Registered: 2018-04-30
Posts: 19

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

Thanks for your quick answer!

I was trying to have a replica before upgrading :-) I'll go with scp, then upgrade, then XSIBackup doing our regular replicas off-site.

Offline

#4 2021-07-28 10:30:17

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

Did '/vmfs/volumes/Backups/backups_ctec' exist in the remote FS?
It all seems related to an inexistent remote dir, although it should have been created previously.

You may encounter problems when trying to create an SSH tunnel between too distant versions of OpenSSH, as most if not all KEX algorithms and ciphers may have been deprecated.

In this case it looks like they were able to agree on some minimum common denominator in terms of KEX & ciphers, as the remote binary could be updated.

Offline

#5 2021-07-30 07:18:22

underdpt
Member
Registered: 2018-04-30
Posts: 19

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

Yeah, it exists in the remote FS. Indeed, XSIBackup did create it on first backup test. Also the backup somehow works, it seems to fail when removing the CTK info or renaming some files. This is the content of one VM backup dir:

[root@REDACTED:/vmfs/volumes/5ba26f3e-372b4cf8-8bb0-ac1f6b16e426/backups_ctec/Lecturas] ls -la
total 75598464
drwxr-xr-x    1 root     root         77824 Jul 24 09:03 .
drwxr-xr-x    1 root     root         73728 Jul 23 19:54 ..
drwxr-xr-x    1 root     root         77824 Jul 24 09:03 .map
-rw-r--r--    1 root     root      17190912 Jul 24 09:01 Lecturas-000002-delta.vmdk
-rw-r--r--    1 root     root           387 Jul 24 09:01 Lecturas-000002.vmdk
-rw-r--r--    1 root     root         31777 Jul 23 18:38 Lecturas-Snapshot1.vmsn
-rw-r--r--    1 root     root         31777 Jul 24 07:28 Lecturas-Snapshot2.vmsn
-rw-r--r--    1 root     root         31784 Jul 24 09:02 Lecturas-Snapshot3.vmsn
-rw-r--r--    1 root     root     214748364800 Jul 24 08:58 Lecturas-flat.vmdk
-rw-r--r--    1 root     root          8684 Jul 24 09:01 Lecturas.nvram
-rw-r--r--    1 root     root           557 Jul 24 09:00 Lecturas.vmdk
-rw-r--r--    1 root     root           483 Jul 24 08:47 Lecturas.vmsd
-rw-r--r--    1 root     root            77 Jul 24 09:01 Lecturas.vmsd.tmp
-rw-r--r--    1 root     root          3262 Jul 24 08:47 Lecturas.vmx
-rw-r--r--    1 root     root          3262 Jul 24 09:01 Lecturas.vmx.tmp
-rw-r--r--    1 root     root          3263 Jul 24 08:47 Lecturas.vmxf
-rw-r--r--    1 root     root          3262 Jul 24 09:01 Lecturas.vmx~
-rw-r--r--    1 root     root        605141 Jul 24 09:01 vmware-1.log
-rw-r--r--    1 root     root        177153 Jul 24 09:01 vmware-2.log
-rw-r--r--    1 root     root        169246 Jul 24 09:01 vmware-3.log
-rw-r--r--    1 root     root       3634137 Jul 24 09:01 vmware-4.log
-rw-r--r--    1 root     root        443310 Jul 24 09:01 vmware.log

Offline

#6 2021-07-30 07:23:02

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

We aren't quite getting you.
Why do you expect the CBT backup to work when you eliminate the .ctk files?
Why do you expect a --replica to continue to work when you rename files manually?

It's like removing the wheels of your car and then complaining because it doesn't move.
Asuming the rename was done right (that's a different story) you must completely reinitialize the replica.
To reset the CTK files you have the --reset-cbt argument.

Offline

#7 2021-07-30 07:44:21

underdpt
Member
Registered: 2018-04-30
Posts: 19

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

I'm sorry you didn't understood me (maybe my English isn't so good).

I didn't rename or remove anything, I'm talking about the messages logged on the backup:

...
All snapshots were removed, as Lecturas is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T15:56:43 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) Lecturas_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Lecturas
....
Total size:                                                                    200.00 GB    | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:25 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:26 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-23T18:28:26 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-23T18:27:18

So, to synthetize:

- I'm trying to backup a ESXi 5.5 host
- XSIBackup tries to remove a VM that didn't exist (Lecturas_XSIREP). I never ran the backup with the argument --options=R
- XSIBackup tries to remove CTK Info from .vmx.tmp and from .vmdk file descriptors / descriptor hashes, and it fails due to a syntax error
- XSIBackup tries to rename .vmx.tmp file to .vmx and .vmsd.tmp file to .vmsd, that fails due to a syntax error

I'm going to try an scp copy and then upgrade the host, but it felt that you should know about these, so that's why I did the report.

Offline

#8 2021-07-30 14:30:45

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

What is the job you are executing?.

You have some kind of bash error. We have tested the software today from ESXi 5.5 to 6.5 with no issue.
Prepend strace to your job to get some detailed system call level output.

strace ./xsibackup ...

Offline

#9 2021-07-30 17:55:43

underdpt
Member
Registered: 2018-04-30
Posts: 19

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

Hello,

This is the command:

strace ./xsibackup --replica=cbt "VMs(Lecturas)" root@REDACTED:22:/vmfs/volumes/Backups/backups_ctec mail-to=REDACTED --subject="Backup ESXi CTEC" --html-template=001 1> /vmfs/volumes/datastore1/xsibackup_strace.log 2>&1

I'm unable to send the full strace log, I tried to remove some parts to 340kb but the forum throws an 404 error.

You can download it from:

https://cloud.mainsip.com/index.php/s/5e7fe5tvdQqRjJo

If you need the original log, please give me an email address as I don't want to put it on a public forum.

Thanks for your support.

Offline

#10 2021-07-31 09:56:54

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

What is this?: "> 1"
Why is the --mail-to argument missing the first "--"?

You are most probably generating your own errors by using a malformed job syntax.
In fact we believe it's quite remarkable that the job took place in the end.

Please, run this and post the output

./xsibackup --replica=cbt "VMs(Lecturas)" root@REDACTED:22:/vmfs/volumes/Backups/backups_ctec

Offline

#11 2021-07-31 12:55:49

underdpt
Member
Registered: 2018-04-30
Posts: 19

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

Hello,

">1" is to send all strace output to a file.

"--mail-to" is missing the "--" due to a copy/paste error.

This is the output of the command:

# ./xsibackup --replica=cbt "VMs(Lecturas)" root@REDACTED:22:/vmfs/volumes/Backups/backups_ctec
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.5.0.11: Backup & Replication Software                   |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 5 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
License: 000306A90000000000000000e06995d797e0 | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
Remote system: ESXi
-------------------------------------------------------------------------------------------------------------
PID: 524375708, Running job as: root
-------------------------------------------------------------------------------------------------------------
Remote xsibackup binary found at: /scratch/XSI/XSIBackup-DC/xsibackup
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/Backups/backups_ctec
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/Backups/backups_ctec'
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:01:55 | Error code 1500 at file common.c, line 1500 | Error description: could not get remote FS available space at /vmfs/volumes/Backups/backups_ctec, details: No such file or directory
-------------------------------------------------------------------------------------------------------------
Available space in backup volume:           -1 (17179.87 PB)
-------------------------------------------------------------------------------------------------------------
Lecturas Hardware Version is: 8
-------------------------------------------------------------------------------------------------------------
All snapshots were removed, as Lecturas is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:01:57 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) Lecturas_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Lecturas
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : Lecturas (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: Lecturas
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-07-31T10:02:10
-------------------------------------------------------------------------------------------------------------
2021-07-31 10:02:10 | Backing up 23 files, total size is 204.12 GB
-------------------------------------------------------------------------------------------------------------
    NUMBER                                                         FILE             SIZE          PROGRESS
-------------------------------------------------------------------------------------------------------------
    1/23                                                   Lecturas.vmx          3.19 KB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    2/23                                                  Lecturas.vmxf          3.19 KB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    3/23                                                  Lecturas.vmsd        446.00 B     | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    4/23                                   Lecturas-flat.vmdk (CBT 147)        200.00 GB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 204800 out of 204800                                    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
    5/23                                                  Lecturas.vmdk        557.00 B     | Done  97.98%
-------------------------------------------------------------------------------------------------------------
    6/23                                                   vmware-1.log        590.96 KB    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
    7/23                                                   vmware-3.log        165.28 KB    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
    8/23                                                 Lecturas.nvram          8.48 KB    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
    9/23                                                   vmware-2.log        173.00 KB    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
   10/23                                                   vmware-4.log          3.47 MB    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
   11/23                                 vmx-Lecturas-1993332819-1.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   12/23                                               Lecturas.vmx.lck                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   13/23                                                  Lecturas.vmx~          3.18 KB    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
   14/23                                                     vmware.log          1.05 MB    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
   15/23                                         Lecturas-76cfd853.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   16/23                                              Lecturas-ctk.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   17/23                                              Lecturas.vmsd.tmp         44.00 B     | Done  97.98%
-------------------------------------------------------------------------------------------------------------
   18/23                                               Lecturas.vmx.tmp          3.18 KB    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
   19/23                                               Lecturas-aux.xml         13.00 B     | Done  97.98%
-------------------------------------------------------------------------------------------------------------
   20/23                                       Lecturas-Snapshot13.vmsn         31.03 KB    | Done  97.98%
-------------------------------------------------------------------------------------------------------------
   21/23                                     Lecturas-000001-delta.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   22/23                                           Lecturas-000001.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   23/23                                       Lecturas-000001-ctk.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
Total size:                                                                    200.01 GB    | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:47 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:47 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:47 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:47 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:09:48 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-31T10:08:51
-------------------------------------------------------------------------------------------------------------
Time taken: 00:06:41 (401 sec.)
-------------------------------------------------------------------------------------------------------------
Total time:      401 sec.
-------------------------------------------------------------------------------------------------------------
Full file speed:                                                                           510.74 mb/s
-------------------------------------------------------------------------------------------------------------
Real data speed:                                                                            11.18 mb/s
-------------------------------------------------------------------------------------------------------------
Item backup completed with errors
-------------------------------------------------------------------------------------------------------------
Final checksum: 2188883130 bytes were sent and confirmed to have been written remotely
-------------------------------------------------------------------------------------------------------------
2021-07-31T10:09:49 | Error code 4574 at file xsibackup.c, line 4574 | Error description: some error/s were raised while backing up: VMs(Lecturas), error count is: 7
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Unlocked backup               OK
-------------------------------------------------------------------------------------------------------------

I also removed the destination folder to be sure there's nothing in it causing the error. This is the result after removing the folder:

# ./xsibackup --replica=cbt "VMs(Lecturas)" root@REDACTED:22:/vmfs/volumes/Backups/backups_ctec
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.5.0.11: Backup & Replication Software                   |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 5 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
License: 000306A90000000000000000e06995d797e0 | (c)XSIBackup-DC
-------------------------------------------------------------------------------------------------------------
Remote system: ESXi
-------------------------------------------------------------------------------------------------------------
PID: 524377608, Running job as: root
-------------------------------------------------------------------------------------------------------------
Remote xsibackup binary found at: /scratch/XSI/XSIBackup-DC/xsibackup
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /vmfs/volumes/Backups/backups_ctec
-------------------------------------------------------------------------------------------------------------
Performing --replica action
-------------------------------------------------------------------------------------------------------------
Backup folder '/vmfs/volumes/Backups/backups_ctec'
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:11:24 | Error code 1500 at file common.c, line 1500 | Error description: could not get remote FS available space at /vmfs/volumes/Backups/backups_ctec, details: No such file or directory
-------------------------------------------------------------------------------------------------------------
Available space in backup volume:           -1 (17179.87 PB)
-------------------------------------------------------------------------------------------------------------
Lecturas Hardware Version is: 8
-------------------------------------------------------------------------------------------------------------
All snapshots were removed, as Lecturas is engaged in a CBT job
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T10:11:26 | Error code 3477 at file xsibackup.c, line 3477 | Error description: could not delete snapshot files for VM(0) Lecturas_XSIREP, error:
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Lecturas
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : Lecturas (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Remote target dir is empty, full sync will be performed
-------------------------------------------------------------------------------------------------------------
Virtual Machine: Lecturas
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-07-31T10:11:40
-------------------------------------------------------------------------------------------------------------
2021-07-31 10:11:40 | Backing up 23 files, total size is 204.14 GB
-------------------------------------------------------------------------------------------------------------
    NUMBER                                                         FILE             SIZE          PROGRESS
-------------------------------------------------------------------------------------------------------------
    1/23                                                   Lecturas.vmx          3.19 KB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    2/23                                                  Lecturas.vmxf          3.19 KB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    3/23                                                  Lecturas.vmsd        447.00 B     | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    4/23                                Lecturas-flat.vmdk (CBT 1 full)        200.00 GB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 204800 out of 204800                                    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    5/23                                                  Lecturas.vmdk        557.00 B     | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    6/23                                                   vmware-1.log        590.96 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    7/23                                                   vmware-3.log        165.28 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    8/23                                                 Lecturas.nvram          8.48 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
    9/23                                                   vmware-2.log        173.00 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   10/23                                                   vmware-4.log          3.47 MB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   11/23                                 vmx-Lecturas-1993332819-1.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   12/23                                               Lecturas.vmx.lck                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   13/23                                                  Lecturas.vmx~          3.18 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   14/23                                                     vmware.log          1.08 MB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   15/23                                         Lecturas-76cfd853.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   16/23                                              Lecturas-ctk.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   17/23                                              Lecturas.vmsd.tmp         44.00 B     | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   18/23                                               Lecturas.vmx.tmp          3.18 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   19/23                                               Lecturas-aux.xml         13.00 B     | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   20/23                                       Lecturas-Snapshot14.vmsn         31.03 KB    | Done  97.97%
-------------------------------------------------------------------------------------------------------------
   21/23                                     Lecturas-000001-delta.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   22/23                                           Lecturas-000001.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   23/23                                       Lecturas-000001-ctk.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
Total size:                                                                    200.01 GB    | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:23 | Error code 4287 at file xsibackup.c, line 4287 | Error description: could not remove CTK Info from .vmx.tmp file, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:23 | Error code 4301 at file xsibackup.c, line 4301 | Error description: could not remove CTK info from .vmdk file descriptors, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:24 | Error code 4316 at file xsibackup.c, line 4316 | Error description: could not update the .vmdk file descriptors hashes, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:24 | Error code 4334 at file xsibackup.c, line 4334 | Error description: could not rename .vmx.tmp file to .vmx, error:
-------------------------------------------------------------------------------------------------------------
sh: syntax error: unterminated quoted string
2021-07-31T12:36:24 | Error code 4343 at file xsibackup.c, line 4343 | Error description: could not rename .vmsd.tmp file to .vmsd, error:
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-07-31T12:35:34
-------------------------------------------------------------------------------------------------------------
Time taken: 02:23:54 (8634 sec.)
-------------------------------------------------------------------------------------------------------------
Total time:     8634 sec.
-------------------------------------------------------------------------------------------------------------
Full file speed:                                                                            23.72 mb/s
-------------------------------------------------------------------------------------------------------------
Real data speed:                                                                             9.67 mb/s
-------------------------------------------------------------------------------------------------------------
Item backup completed with errors
-------------------------------------------------------------------------------------------------------------
Final checksum: 77400130812 bytes were sent and confirmed to have been written remotely
-------------------------------------------------------------------------------------------------------------
2021-07-31T12:36:25 | Error code 4574 at file xsibackup.c, line 4574 | Error description: some error/s were raised while backing up: VMs(Lecturas), error count is: 7
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Unlocked backup               OK
-------------------------------------------------------------------------------------------------------------

Offline

#12 2021-08-01 00:06:26

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

It is clear that you can't execute shell commands in the remote system. Make sure that you haven't activated secure boot and that the shell is working, you may need to reinstall ESXi.

There have been shell bugs in the past, try to upgrade your remote ESXi system to the latest patch in the branch.

Offline

#13 2021-08-02 06:30:46

underdpt
Member
Registered: 2018-04-30
Posts: 19

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

Hello,

First, I wanna thank you for your kind support. Much appreciated!

FYI, the remote host is hosting backups from serveral systems correctly, all from XSIBackup, so I think the remote host isn't involved in this problem. All the other hosts doing backups to this same remote host are EXSi 6.5 or higher, that's why I though the ESXi version of the system was involved on the error.

Offline

#14 2021-08-02 10:42:11

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

It could be, we have nonetheless reproduced your environment 5.5 to 6.5 without issue.
Post the output of

vmware -v

From both hosts and we will try to reproduce the exact same environment accurate to the build number.
In any case the problem is rather clear. From your strace.

pipe([10, 11])                          = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x32000910) = 524212574
close(11)                               = 0
fstat(10, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3ffc0ec7000
read(10, sh: syntax error: unterminated quoted string
"", 600) 

The pipe command returns a shell syntax error that isn't happening in other builds nor in our tests. Everything points at some bug in the OpenSSH client.

This is the exact translated command that you can use to debug your scenario and the one that returns the first error. Run from your ESXi 5.5 host and post the output.

ssh -vvv -i xsibackup_id_rsa root@REDACTED "\"/scratch/XSI/XSIBackup-DC/xsibackup\" --free-space \"/vmfs/volumes/Backups/backups_ctec\""

(*) The -vvv will just make ssh client to be verbose.

The problem seems to be related to the double quote escaping, most probably the ssh client is not parsing them well, which would lead us to our very first answer: patch your ESXi 5.5 host.

If you want to make a quick test, just copy some ssh client binary from some close version, maybe the one packed in ESXi 6.0 should work. Still that would not survive a reboot, which would again lead you to the need to patch your host to the latest build in the 5.5 branch.

Offline

#15 2021-08-02 17:10:53

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

We have detected this issue in some other builds, it has to do with the same we comment above. We'll try to work the issue around by running shell commands in some alternative way.

UPDATE:

We can confirm this is due to a bug in the OpenSSL implementation shipped with some versions of (c)ESXi.
The bug consists in that the message "WARNING: can't open config file: /usr/ssl/openssl.cnf" is returned every time you invoke the openssl binary, it has caused some other issues in the past.

We'll work this issue around and publish as 1.5.0.12

Offline

#16 2021-08-02 23:31:19

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Issue on ESXi 5.5: sh: syntax error: unterminated quoted string

The problem is solved in 1.5.0.12 available in the download zone of the user area.
Please confirm whether this fixed your issue.

Offline

Board footer