©XSIBackup-Free: Free Backup Software for ©VMWare ©ESXi

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software


You are not logged in.

#1 2021-06-18 02:48:38

jakorsme
Member
Registered: 2021-06-18
Posts: 12

Backup blew up after 30%

I had been really excited about this software.

Installed fine, ran backup fine to local drive. Got 30% through the backup and errored out.
I'm running as root - /tmp is 777
/vmfs/volumes/VMDATA/data/33938929867210700.tmp does not exist.
/vmfs/volumes/VMDATA/data/ is root:root 755

2021-06-17T22:03:20 | Error code 209 at file signal.c, line 209 | Error description: raised SIGTERM (2) in job, num of errors: 2, check error.log
2021-06-17T22:03:20 | Error code 214 at file signal.c, line 214 | Error description: raised SIGTERM (2) (2) in job, total errors: 2, check error.log
2021-06-17T22:03:55 | Error code 1058 at file common.c, line 1058 | Error description: .lock on '/tmp/xsibackup.pid' after 30 s
2021-06-17T22:03:55 | Error code 133 at file signal.c, line 133 | Error description: can't remove PID from /tmp/xsibackup.pid
2021-06-18T01:57:57 | Error code 389 at file dedup-in.c, line 389 | Error description: can't open temp block file: /vmfs/volumes/VMDATA/data/33938929867210700.tmp
2021-06-18T01:57:57 | Error code 214 at file signal.c, line 214 | Error description: raised SIGTERM (11) (28) in job, total errors: 11, check error.log

Offline

#2 2021-06-19 08:42:21

admin
Administrator
Registered: 2017-04-21
Posts: 1,821

Re: Backup blew up after 30%

Please post the whole job and output. Knowing the command that you have run (c)ESXi version, kind of DS, file system, etc.. is crucial to offering an answer.

Nonetheless you have a bunch of errors, your output denotes some kind of gross error in what you are doing. Are you sure that your datastore is there (VMDATA)? and that it's not full. Is it VMFS-5 by some chance?
You can't perform a deduplicated backup to a VMFS-5 datastore, it only has 130,000 available inodes, not enough to host deduplicated data.

You may very well replicate to it on the other side.

Use a fast file system over NFS to host deduplicated repositories: ext4 or XFS are your best bet. If you can use SSD HDs even better, take on account that deduplication will save 98% of your space once you add a number of restore points, thus using some SSD about double the size of a single deduplicated backup set will be able to host hundreds of restore points.

Offline

#3 2021-06-19 20:10:43

jakorsme
Member
Registered: 2021-06-18
Posts: 12

Re: Backup blew up after 30%

Yes it is vmfs-5. Outputting locally to VMDATA (it is only 4% full). I'm trialing so I can only output to the local drive (my understanding).
It is backing up a smaller vm (80G thin) ok to same location. Made me think it was maybe your trial limit doing this. Thinking it may be something else I didn't want to just license the trial and have it not resolve the issue. Also when this failed, I tried to run a smaller vm backup and it failed with "out of disk space" error, even though there was still almost 4TB of space available (maybe out of inodes is throwing same exception). I could only backup another vm after deleting the repository from the failed backup.
Currently trying again with the ouput going to an nfs mounted nas.

[root@elevation:/vmfs/volumes/588366f8-884c8d8c-ff51-902b3451b465/XSI/XSIBackup-DC] ./xsibackup --backup "VMs(Black)" /vmf
s/volumes/VMDATA/Backups
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-Free 1.5.0.8: Backup & Replication Software                  |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
License: unlicensed trial version, remaining trial time: 31:35:42 | (c)XSIBackup-Free
-------------------------------------------------------------------------------------------------------------
PID: 1898025, Running job as: root
-------------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-------------------------------------------------------------------------------------------------------------
Block size is 1.00 MB (1048576 bytes)
-------------------------------------------------------------------------------------------------------------
Performing --backup action
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-Free setting repository at /vmfs/volumes/VMDATA/Backups
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
Black Hardware Version is: 13
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Black
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : Black (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: Black
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-06-19T04:24:23
-------------------------------------------------------------------------------------------------------------
2021-06-19 04:24:23 | Backing up 23 files, total size is 162.24 GB
-------------------------------------------------------------------------------------------------------------
    NUMBER                                                       FILE             SIZE          PROGRESS
-------------------------------------------------------------------------------------------------------------
    1/23                                                      Black.vmx          3.30 KB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    2/23                                              Black_0-flat.vmdk        150.00 GB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 19.48% done | block 29917 out of 153600                                      | Done  18.01%2021-06-19T06:39:11 | Error code 389 at file dedup-in.c, line 389 | Error description: can't open temp block file: /vmfs/volumes/VMDATA/Backups/data/34269407511556112.tmp
-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------
2021-06-19T06:39:11 | Error code 214 at file signal.c, line 214 | Error description: raised SIGTERM (11) (28) in job, total errors: 11, check error.log
-------------------------------------------------------------------------------------------------------------
Removed current target dir: /vmfs/volumes/5894f2f4-1995ab20-7898-902b3451b465/Backups/20210619042418/Black
-------------------------------------------------------------------------------------------------------------
SIGTERM (11) condition was trapped: check logs for more details
-------------------------------------------------------------------------------------------------------------
Cleaning up...
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Unlocked backup               OK
-------------------------------------------------------------------------------------------------------------
Log file unmapped             OK
-------------------------------------------------------------------------------------------------------------
Removed PID                   OK
-------------------------------------------------------------------------------------------------------------
[root@elevation:/vmfs/volumes/588366f8-884c8d8c-ff51-902b3451b465/XSI/XSIBackup-DC]

Last edited by jakorsme (2021-06-19 23:00:41)

Offline

#4 2021-06-20 05:35:41

jakorsme
Member
Registered: 2021-06-18
Posts: 12

Re: Backup blew up after 30%

I got the backup to work by outputting to a nas.

However after finishing the backup, when I try to do an --info on the repository. No matter how I escape the spaces in the path (double or single quote enclosed or back-slash escaped spaces) --info blows up. Apparently the code isn't passing the path with spaces as one unit.

>>> I renamed the mount to NAS1 to eliminate the spaces and --info now works. But it should really be able to handle names with spaces if escaped or quoted. Right?

Here's the output. I also included the original --backup command output afterwards:
-------------------------------------------Info----------------------------------------

[root@elevation:/vmfs/volumes/588366f8-884c8d8c-ff51-902b3451b465/XSI/XSIBackup-DC] ./xsibackup --info /vmfs/volumes/nas\ 1\ john
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-Free 1.5.0.8: Backup & Replication Software                  |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
PID: 1909763, Running job as: root
-------------------------------------------------------------------------------------------------------------
Deduplicated repository information
-------------------------------------------------------------------------------------------------------------
Compression: 1
Block size: 1048576
Block count: 115476
-------------------------------------------------------------------------------------------------------------
Retrieving block information...
-------------------------------------------------------------------------------------------------------------
sh: can't create /vmfs/volumes/nas: Operation not permitted
sort: /vmfs/volumes/nas: No such file or directory
2021-06-20T05:15:53 | Error code 3577 at file common.c, line 3577 | Error description: error ordering .blocklog file: /vmfs/volumes/nas 1 john/data/.blocklog
-------------------------------------------------------------------------------------------------------------
2021-06-20T05:15:53 | Error code 67 at file info.c, line 67 | Error description: error ordering .blocklog file at: /vmfs/volumes/nas 1 john/data/.blocklog
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
[root@elevation:/vmfs/volumes/588366f8-884c8d8c-ff51-902b3451b465/XSI/XSIBackup-DC]


-------------------------------------------Orig bckup --------------------------------

./xsibackup --backup "VMs(Black)" "/vm
fs/volumes/nas 1 john"
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-Free 1.5.0.8: Backup & Replication Software                  |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
License: unlicensed trial version, remaining trial time: 15:37:32 | (c)XSIBackup-Free
-------------------------------------------------------------------------------------------------------------
PID: 1905571, Running job as: root
-------------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-------------------------------------------------------------------------------------------------------------
Block size is 1.00 MB (1048576 bytes)
-------------------------------------------------------------------------------------------------------------
Performing --backup action
-------------------------------------------------------------------------------------------------------------
(c)XSIBackup-Free setting repository at /vmfs/volumes/nas 1 john
-------------------------------------------------------------------------------------------------------------
Remote 'data' folder does not exist, creating it...
-------------------------------------------------------------------------------------------------------------
Item number 1 in this job
-------------------------------------------------------------------------------------------------------------
Black Hardware Version is: 13
-------------------------------------------------------------------------------------------------------------
Virtual Machine Name: Black
-------------------------------------------------------------------------------------------------------------
Creating snapshot VM : Black (powered on)
-------------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-------------------------------------------------------------------------------------------------------------
Virtual Machine: Black
-------------------------------------------------------------------------------------------------------------
Backup start date: 2021-06-19T20:22:33
-------------------------------------------------------------------------------------------------------------
2021-06-19 20:22:33 | Backing up 23 files, total size is 162.24 GB
-------------------------------------------------------------------------------------------------------------
    NUMBER                                                       FILE             SIZE          PROGRESS
-------------------------------------------------------------------------------------------------------------
    1/23                                                      Black.vmx          3.30 KB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
    2/23                                              Black_0-flat.vmdk        150.00 GB    | Done   0.00%
-------------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block 153600 out of 153600                                    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
    3/23                                                   Black_0.vmdk        551.00 B     | Done  92.46%
-------------------------------------------------------------------------------------------------------------
    4/23                                                     Black.vmsd        435.00 B     | Done  92.46%
-------------------------------------------------------------------------------------------------------------
    5/23                                                    Black.nvram          8.48 KB    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
    6/23                                                     Black.vmx~          3.29 KB    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
    7/23                                                     Black.vmxf          3.08 KB    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
    8/23                                                   vmware-2.log        463.94 KB    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
    9/23                                                   vmware-3.log        983.66 KB    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
   10/23                                                   vmware-4.log          1.23 MB    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
   11/23                                                   vmware-7.log          2.79 MB    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
   12/23                                                   vmware-6.log        508.62 KB    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
   13/23                                    vmx-Black-1838801243-2.vswp        110.00 MB    | Done  92.46%
-------------------------------------------------------------------------------------------------------------
   14/23                                                   vmware-5.log        347.73 KB    | Done  92.53%
-------------------------------------------------------------------------------------------------------------
   15/23                                vmx-Black-1838801243-1.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   16/23                                              Black.vmx.lck                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   17/23                                                     vmware.log        546.27 KB    | Done  92.53%
-------------------------------------------------------------------------------------------------------------
   18/23                                        Black-6d99e15b.vswp                    [open excluded]
-------------------------------------------------------------------------------------------------------------
   19/23                                                 Black.vmsd.tmp         43.00 B     | Done  92.53%
-------------------------------------------------------------------------------------------------------------
   20/23                                                  Black.vmx.tmp          3.29 KB    | Done  92.53%
-------------------------------------------------------------------------------------------------------------
   21/23                                           Black-Snapshot3.vmsn         31.04 KB    | Done  92.53%
-------------------------------------------------------------------------------------------------------------
   22/23                                  Black_0-000001-delta.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
   23/23                                        Black_0-000001.vmdk                 [skipped excluded]
-------------------------------------------------------------------------------------------------------------
Total size:                                                                    150.11 GB    | Done 100.00%
-------------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-------------------------------------------------------------------------------------------------------------
Backup end date: 2021-06-20T04:28:36
-------------------------------------------------------------------------------------------------------------
Time taken: 08:06:03 (29163 sec.)
-------------------------------------------------------------------------------------------------------------
Total time:    29163 sec.
-------------------------------------------------------------------------------------------------------------
Full file speed:                                                                         5.27 mb/s
-------------------------------------------------------------------------------------------------------------
Real data speed:                                                                         7.77 mb/s
-------------------------------------------------------------------------------------------------------------
Item backup completed without errors
-------------------------------------------------------------------------------------------------------------
Differential blocks were added to the .blocklog database
-------------------------------------------------------------------------------------------------------------
Data processing completed successfully
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Unlocked backup               OK
-------------------------------------------------------------------------------------------------------------
Removed PID                   OK
-------------------------------------------------------------------------------------------------------------

Last edited by jakorsme (2021-06-20 06:10:39)

Offline

#5 2021-06-20 07:18:35

jakorsme
Member
Registered: 2021-06-18
Posts: 12

Re: Backup blew up after 30%

More problems with --info and apparently the backup that said it was successful without errors.

[root@elevation:/vmfs/volumes/588366f8-884c8d8c-ff51-902b3451b465/XSI/XSIBackup-DC] ./xsibackup --info /vmfs/volumes/NAS1
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-Free 1.5.0.8: Backup & Replication Software                  |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 5 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
PID: 1910182, Running job as: root
-------------------------------------------------------------------------------------------------------------
Deduplicated repository information
-------------------------------------------------------------------------------------------------------------
Compression: 1
Block size: 1048576
Block count: 115476
-------------------------------------------------------------------------------------------------------------
Retrieving block information...
-------------------------------------------------------------------------------------------------------------
2021-06-20T06:53:39 | Error code 165 at file info.c, line 165 | Error description: can't get blockhash, bad .blocklog format?, details: No such file or directory

-------------------------------------------------------------------------------------------------------------
2021-06-20T06:53:39 | Error code 174 at file info.c, line 174 | Error description: can't get blocksize, bad .blocklog format?, details: No such file or directory

-------------------------------------------------------------------------------------------------------------
2021-06-20T06:53:39 | Error code 174 at file info.c, line 174 | Error description: can't get blocksize, bad .blocklog format?, details: No such file or directory

<<<<<
<<<<<Hundreds of these lines>>>>>
<<<<<


-------------------------------------------------------------------------------------------------------------
2021-06-20T06:53:59 | Error code 174 at file info.c, line 174 | Error description: can't get blocksize, bad .blocklog format?, details: No such file or directory

-------------------------------------------------------------------------------------------------------------
2021-06-20T06:53:59 | Error code 174 at file info.c, line 174 | Error description: can't get blocksize, bad .blocklog format?, details: No such file or directory

-------------------------------------------------------------------------------------------------------------
2021-06-20T06:53:59 | Error code 174 at file info.c, line 174 | Error description: can't get blocksize, bad .blocklog format?, details: No such file or directory

-------------------------------------------------------------------------------------------------------------
2021-06-20T06:53:59 | Error code 174 at file info.c, line 174 | Error description: can't get blocksize, bad .blocklog format?, details: No such file or directory

-------------------------------------------------------------------------------------------------------------
2021-06-20T06:53:59 | Error code 174 at file info.c, line 174 | Error description: can't get blocksize, bad .blocklog format?, details: No such file or directory

-------------------------------------------------------------------------------------------------------------
                    |          Real data   |            Hosted data   |
-------------------------------------------------------------------------------------------------------------
Block number        |             115476   |                    679   |
Data volume         |           51.37 GB   |              671.84 MB   |
Compression         |             -7729.73%   |                      -   |
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
[root@elevation:/vmfs/volumes/588366f8-884c8d8c-ff51-902b3451b465/XSI/XSIBackup-DC]

Offline

#6 2021-06-20 10:56:40

admin
Administrator
Registered: 2017-04-21
Posts: 1,821

Re: Backup blew up after 30%

Do not even try to store a deduplicated backup on VMFS-5, we won't comment any further around this issue, we already explained on that.

You are running --info on a datastore, why?. It is meant to be run on a backup repository.
Please read the manual and examples before using the software.
(c)XSIBackup-DC

We will modify the info.c module so that it terminates with a clear message when no .blocklog file is detected. Nonetheless we can't detect every possible wrong way that users will be trying to use the software to print a message stating why it shouldn't be used that way.

The job below is conceptually wrong:

./xsibackup --backup "VMs(Black)" "/vmfs/volumes/nas 1 john"

You should not backup to a repository in the root of a datastore.
You should instead run:

./xsibackup --backup "VMs(Black)" "/vmfs/volumes/nas 1 john/some_repository"

And then get information on that repository running

./xsibackup --info "/vmfs/volumes/nas 1 john/some_repository"

We specifically block attempts to backup to the root of a datastore. Maybe the fact that you used some datastore containing spaces managed to work those checks around, we'll check that point.

Needless to say you should avoid spaces in paths as well as special characters or accented ones.

Offline

#7 2021-06-20 22:56:13

jakorsme
Member
Registered: 2021-06-18
Posts: 12

Re: Backup blew up after 30%

It worked to add a repo after the nfs mount point (/vmfs/volumes/NAS1/elevation_repo) to successfully backup and info the smaller VM (couldn't try my larger VM again because my trial just expired).

Offline

#8 2021-06-21 08:08:41

admin
Administrator
Registered: 2017-04-21
Posts: 1,821

Re: Backup blew up after 30%

There is already a new release with a new expiration date, download and overwrite your old main xsibackup binary or install on top.

Offline

Board footer