©XSIBackup-Free: Free Backup Software for ©VMWare ©ESXi

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software


You are not logged in.

#1 2020-12-16 09:34:57

wowbagger
Member
Registered: 2017-05-11
Posts: 41

VMWare ESXi: hardware Segmentation fault

Running this cmdline:

/scratch/xsibackup/xsibackup --backup "VMs(ansible01.infra.dileoz.network,vlpr-mongodb01.prd.saas.dpp.dileoz.network,pfsense2_dileoz_network,vlpr-jasper01.prd.saas.dpp.dileoz.network,vwac03_acc_template_empower_dileoz_network,sftp.prd.saas.dileoz.network,vwac02_acc_pov_empower_dileoz_network,vwac01_acc_saas_empower_dileoz_network,vwpr02_prd_pov_empower_dileoz_network,vlpr-order01.prd.saas.dpp.dileoz.network,vcenter01_infra_dileoz_network,codeks_prd_knokkeheist_dpp_dileoz_network,DILEWEB0001,Oracle-1,softether.infra.dileoz.network,vault01_dileoz_network,SRVPBSTEST02,SRVPBSSQL02,Graylog2)" /vmfs/volumes/nfs_typhon_olympus/XSI_Backup/2020_DC --verbosity=5 --use-smtp=1 --compression=true --config-backup --subject='Olympus Backup' --mail-to=icc@dileoz.com

Always happens at the exact same VM, number #5.
Backing the VM's individually does not generate the error.

[90m-----------------------------------------------------------------------------------------------------------[0m
<icc@dileoz.com> was evaluated as an e-mail address
[90m|---------------------------------------------------------------------------------|[0m
[90m||-------------------------------------------------------------------------------||[0m
[90m|||[0m[1m   (c)XSIBackup-DC 1.4.2.7: Backup & Replication Software                   [0m [90m|||[0m
[90m|||[0m[1m   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved[0m    [90m|||[0m
[90m||-------------------------------------------------------------------------------||[0m
[90m|---------------------------------------------------------------------------------|[0m
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
[90m|---------------------------------------------------------------------------------|[0m
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
[90m-----------------------------------------------------------------------------------------------------------[0m
License: [1m00050654000000000000000090b11c1f6861[0m
[90m-----------------------------------------------------------------------------------------------------------[0m
Primary TMP folder: /tmp/xsi/2789239
[90m-----------------------------------------------------------------------------------------------------------[0m
Secondary TMP folder: /scratch/xsibackup/tmp/2789239
[90m-----------------------------------------------------------------------------------------------------------[0m
PID: 2789239, Running job as: [1mroot[0m
[90m-----------------------------------------------------------------------------------------------------------[0m
LZJB compression has been enabled
[90m-----------------------------------------------------------------------------------------------------------[0m
Block size is 1.00 MB (1048576 bytes)
[90m-----------------------------------------------------------------------------------------------------------[0m
Performing --backup action
[90m-----------------------------------------------------------------------------------------------------------[0m
Created backup folder at: [90m/vmfs/volumes/nfs_typhon_olympus/XSI_Backup/2020_DC/20201216001541[0m
[90m-----------------------------------------------------------------------------------------------------------[0m
(c)XSIBackup-DC setting repository at /vmfs/volumes/nfs_typhon_olympus/XSI_Backup/2020_DC
[90m-----------------------------------------------------------------------------------------------------------[0m
Located backup folder at: [90m/vmfs/volumes/nfs_typhon_olympus/XSI_Backup/2020_DC/data[0m
[90m-----------------------------------------------------------------------------------------------------------[0m
.xsitools file located: /vmfs/volumes/nfs_typhon_olympus/XSI_Backup/2020_DC/.xsitools
[90m-----------------------------------------------------------------------------------------------------------[0m
Item number 1 in this job
[90m-----------------------------------------------------------------------------------------------------------[0m
Hardware disk: /dev/disks/naa.6d0946600e2ae10026ffec6d9a6d9017
[90m-----------------------------------------------------------------------------------------------------------[0m
Created backup folder at: [90m/vmfs/volumes/4e815752-33eb9462/XSI_Backup/2020_DC/20201216001541/ansible01.infra.dileoz.network[0m
[90m-----------------------------------------------------------------------------------------------------------[0m
Lock file: /vmfs/volumes/4e815752-33eb9462/XSI_Backup/2020_DC/20201216001541/ansible01.infra.dileoz.network/.locked
Virtual Machine: [1mansible01.infra.dileoz.network[0m
[90m-----------------------------------------------------------------------------------------------------------[0m
Backup start date: 2020-12-16T00:15:54
[90m-----------------------------------------------------------------------------------------------------------[0m
2020-12-16 00:15:54 | Backing up 13 files, total size is 108.08 GB
[90m-----------------------------------------------------------------------------------------------------------[0m
    NUMBER                                                       FILE             SIZE          PROGRESS
[90m-----------------------------------------------------------------------------------------------------------[0m

    1/13                   ansible01.infra.dileoz.network-376bec04.hlog        549.00 B     [90m|[0m Done   0.00%
[90m-----------------------------------------------------------------------------------------------------------[0m
FS block size: 1048576
Source file size: 549 bytes
Source file: 0 allocated blocks
Using default block size: 1048576
LZJB compression set by user
[90m-----------------------------------------------------------------------------------------------------------[0m
1 blocks to backup
[90m-----------------------------------------------------------------------------------------------------------[0m
1 lines in .map file vs 1 detected blocks
[90m-----------------------------------------------------------------------------------------------------------[0m

    2/13                   ansible01.infra.dileoz.network-flat.vmdk                    [open excluded]
[90m-----------------------------------------------------------------------------------------------------------[0m

    3/13                            ansible01.infra.dileoz.network.vmdk        552.00 B     [90m|[0m Done   0.00%
[90m-----------------------------------------------------------------------------------------------------------[0m
FS block size: 1048576
Source file size: 552 bytes
Source file: 0 allocated blocks
Using default block size: 1048576
LZJB compression set by user
[90m-----------------------------------------------------------------------------------------------------------[0m
Could not retrieve disk metadata all blocks will be scanned
[90m-----------------------------------------------------------------------------------------------------------[0m
1 blocks to backup
[90m-----------------------------------------------------------------------------------------------------------[0m
1 lines in .map file vs 1 detected blocks
[90m-----------------------------------------------------------------------------------------------------------[0m

    4/13                           ansible01.infra.dileoz.network.nvram          8.48 KB    [90m|[0m Done   0.00%
[90m-----------------------------------------------------------------------------------------------------------[0m
FS block size: 1048576
Source file size: 8684 bytes
Source file: 128 allocated blocks
Using default block size: 1048576
LZJB compression set by user
[90m-----------------------------------------------------------------------------------------------------------[0m
1 blocks to backup
[90m-----------------------------------------------------------------------------------------------------------[0m
1 lines in .map file vs 1 detected blocks
[90m-----------------------------------------------------------------------------------------------------------[0m

    5/13                             ansible01.infra.dileoz.network.vmx          3.48 KB    [90m|[0m Done   0.00%
[90m-----------------------------------------------------------------------------------------------------------[0m


<SNIPPED>


   15/19      vlpr-jasper01.prd.saas.dpp.dileoz.network-Snapshot39.vmsn         19.69 KB    [90m|[0m Done  75.42%
[90m-----------------------------------------------------------------------------------------------------------[0m
FS block size: 1048576
Source file size: 20167 bytes
Source file: 128 allocated blocks
Using default block size: 1048576
LZJB compression set by user
[90m-----------------------------------------------------------------------------------------------------------[0m
1 blocks to backup
[90m-----------------------------------------------------------------------------------------------------------[0m
1 lines in .map file vs 1 detected blocks
[90m-----------------------------------------------------------------------------------------------------------[0m

   16/19     vlpr-jasper01.prd.saas.dpp.dileoz.network-000001-sesparse.vmdk                 [skipped excluded]
[90m-----------------------------------------------------------------------------------------------------------[0m

   17/19      vlpr-jasper01.prd.saas.dpp.dileoz.network-000001.vmdk                 [skipped excluded]
[90m-----------------------------------------------------------------------------------------------------------[0m

   18/19     vlpr-jasper01.prd.saas.dpp.dileoz.network_1-000001-sesparse.vmdk                 [skipped excluded]
[90m-----------------------------------------------------------------------------------------------------------[0m

   19/19     vlpr-jasper01.prd.saas.dpp.dileoz.network_1-000001.vmdk                 [skipped excluded]
[90m-----------------------------------------------------------------------------------------------------------[0m
Total size:                                                                     50.00 GB    [90m|[0m Done 100.00%
[90m-----------------------------------------------------------------------------------------------------------[0m
*** Snapshot was removed ***
[90m-----------------------------------------------------------------------------------------------------------[0m
Total CPU ticks: 173885795048
Avg frequency (Ghz):     2.02
Real data ticks: 437557411326
Real data seconds: 216
Raw data speed:       0.00
Backup end date: 2020-12-16T00:33:21
[90m-----------------------------------------------------------------------------------------------------------[0m
Time taken: 00:01:26 (86 sec.)
[90m-----------------------------------------------------------------------------------------------------------[0m
Total time:      989 sec.
[90m-----------------------------------------------------------------------------------------------------------[0m
Full file speed:                                                                       595.35 mb/s
[90m-----------------------------------------------------------------------------------------------------------[0m
Real data speed:                                                                         0.38 mb/s
[90m-----------------------------------------------------------------------------------------------------------[0m
Item backup completed
[90m-----------------------------------------------------------------------------------------------------------[0m
Item number 5 in this job
[90m-----------------------------------------------------------------------------------------------------------[0m
Hardware dSegmentation fault

What could this mean?

Thx

Offline

#2 2020-12-16 13:52:57

JoJo
Member
Registered: 2019-10-23
Posts: 5

Re: VMWare ESXi: hardware Segmentation fault

I had also Segmentation faults with the newer verison > 1.4.x.x. Without entries in the logs. Then i went back to 1.4.0.0 an no segmention faults at all.

Last edited by JoJo (2020-12-16 13:53:13)

Offline

#3 2020-12-16 16:01:27

wowbagger
Member
Registered: 2017-05-11
Posts: 41

Re: VMWare ESXi: hardware Segmentation fault

Yes, I read somewhere here about a bug in the 1.4.0.0 and an upgrade was probably best, hence why I upgraded. 1.4.0.0 was pretty rock solid indeed. I should have have stayed at 1.4.0.0 wink

It seems the problem is in some way triggered when providing multiple VM's, it never happens when just backing up a single VM.
I'll probably end up looping over the VM's I want backed-up and starting xsibackup with a single VM.

In another run I now got:

2020-12-16T14:12:32 | Error code 131 at file signal.c, line 131 | Error description: some error was trapped SIGTERM (11) (11) while executing the job, error: No space left on device

Which is probably /tmp filling up, but XSIBackup-DC is the only folder with data in /tmp. I have now symlinked /tmp/xsi to another location, perhaps I can get a full set of backups this way.
Maybe I'll have to symlink /scratch/xsibackup/tmp also to another location.

Last edited by wowbagger (2020-12-16 16:03:51)

Offline

#4 2020-12-16 16:33:12

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

JoJo, your problem should be fixed by now, if not contact us for a remote session.
That kind of errors happen when something goes out of control e.g.: a full FS.
There were some circumstances in which the /tmp folders could fill up, we are working to fix that on 1.4.2.8
Please remove the /tmp folder and the local tmp folder at installation root:

rm -rf /tmp/*
rm -rf /scratch/XSIBackup-DC/tmp/*

If the problem persists, just contact our support e-mail to appoint a session too.

UPDATE:

We believe it could just be some variable holding the VM names overflowing, still it halts the backup.
Try to split the VMs in two groups until we find the exact cause. We were about to release 1.4.2.8, we'll wait a bit to check all those implied vars.

Offline

#5 2020-12-16 23:58:44

wowbagger
Member
Registered: 2017-05-11
Posts: 41

Re: VMWare ESXi: hardware Segmentation fault

Splitting did not really work for some reason, perhaps it has to do with the VM name characters. I'm looping over the list so there is just a single xsibackup invocation each time as opposed to providing a long list of VM's.
I noticed in /scratch/downloads/ the exported configs .tgz files accumulate, maybe also a candidate for cleaning after a backup run.

Last edited by wowbagger (2020-12-16 23:59:09)

Offline

#6 2020-12-17 16:51:50

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

Be careful if you cut & paste, you may be inputting characters out of the ASCII set.

Yes, that will need some clean up, those files are rather small and they are deleted on every reboot, but if you use the --config-backup very often, a good number of .tgz files may end up there.

We are working on this issue, we'll come up with a definitive solution or answer after the weekend, if not before.

Offline

#7 2020-12-17 17:02:06

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

Nothing around variables length seems to be the issue. The fact that your VM list is so long made us pointed our investigation in that direction.

We found this message in your log:

Could not retrieve disk metadata all blocks will be scanned

That indicates that some of your VMs reside in an NFS host or that the metadata for that VM is corrupt, as it could not be retrieved.

If you have mixed datastore types VMFS & NFS, try to group VMs by datastore type into two different jobs. We will check that posibility in search of an eventual bug.

If you don't have NFS datastores, then clone that VM with vmkfstools to regenerate the associated metadata.

Offline

#8 2020-12-17 18:24:44

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

wowbagger wrote:

Splitting did not really work for some reason, perhaps it has to do with the VM name characters. I'm looping over the list so there is just a single xsibackup invocation each time as opposed to providing a long list of VM's.
I noticed in /scratch/downloads/ the exported configs .tgz files accumulate, maybe also a candidate for cleaning after a backup run.

Version 1.4.2.8 will include a /scratch/downloads folder clean up.

Offline

#9 2020-12-20 22:28:45

wowbagger
Member
Registered: 2017-05-11
Posts: 41

Re: VMWare ESXi: hardware Segmentation fault

admin wrote:

Could not retrieve disk metadata all blocks will be scanned

That indicates that some of your VMs reside in an NFS host or that the metadata for that VM is corrupt, as it could not be retrieved.

If you have mixed datastore types VMFS & NFS, try to group VMs by datastore type into two different jobs. We will check that posibility in search of an eventual bug.

If you don't have NFS datastores, then clone that VM with vmkfstools to regenerate the associated metadata.

Thanks for your reply.
Yes, it's mixed VMFS/NFS.
For now I'm just running xsibackup for each individual VM but that also fails after a few days with segmentation faults.
The only way to fix it is to give it an empty repository directory, it continues without segmentation faults.
Never had this problem in 1.4.0.0, is there a way to download 1.4.0.0 again?

Offline

#10 2020-12-20 22:56:26

JoJo
Member
Registered: 2019-10-23
Posts: 5

Re: VMWare ESXi: hardware Segmentation fault

I agree with you wowbagger. I have exactly the same behavior as you. Running version 1.4.0.0 without problems so far.

Last edited by JoJo (2020-12-20 22:57:09)

Offline

#11 2020-12-21 08:03:43

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

Contact us to get some older version.

We will revise the code to try to identify your issue, nonetheless it's not a generalized problem and must probably affect some particular condition that you share.

JoJo: you never provided any details on your issue in this forum. Please, do not assume or make others think that you share some common problem when you have not provided details that demonstrate that, it's misleading.

Wowbagger's issue is not even about backup, the segfault is raised when processing information on the VMs, not when backing them up. It's probably something quite trivial, still it will need to be identified to offer a solution.

Please, try to give as much information as you can: whether you are using NFS or VMFS, the size of your VMs, the jobs and output. This goes for JoJo, Wowbagger has already provided enough info.

UPDATE:

Contact us on support and we will provide some special debug version/s with some extra messages that will detect where exactly things are going wrong in your case. Once we detect where the issue is we'll publish the conclusions, otherwise we are going to make this post grow with speculations on the possible cause.

Offline

#12 2020-12-21 10:32:43

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

Just a general note on NFS datastores.
Only VMFS can provide metadata on the VM files to jump over zeroed zones, thus, if you use NFS all file length must be traversed, thus backing up a VM stored in an NFS volume can take several times the lapse it takes to backup the very same VM when it is stored in a VMFS volume.

Offline

#13 2020-12-22 12:53:12

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

UPDATE:

We are setting up some lab environment to try to reproduce your issue. This is an update on what we have found so far:

-    The system call fail collected by strace on your ©ESXi host is clear: ENOSPC (No space left on device). That is: we have hit a system limit.
-    Attached you will find a list of ©ESXi 6.5’s error codes. ENOSPC is used in multiple places, but it’s always related to limit of inodes reached or lack of memory to store file descriptors…
ENOSPC Error
-    There exist multiple bugs in ©ESXi related to that error. In fact ©ESXi 6.7 was buggy at the beginning, we hit some of those bugs while developing ©XSIBackup-Pro.
-    No space left on device errors
-    If we are able to reproduce the issue on multiple ©ESXi versions, that means we have hit some hardcoded limit and that we need to rethink how some parts DC work to allow it to go further, as it did up to 1.4.0.0. In newer versions we improved indexing of the .blocklog file by adding a more complex dynamic indexing system. That makes DC faster, but it is more resource intensive. If on the contrary the issue is only reproduceable in your ©ESXi build, then that would mean that you just hit an ©ESXi bug and an upgrade would solve the issue.

Also:

Recheck that your NFS bond is working properly and that you are not storing your backup repositories in the root FS instead of in the NFS server, just in case.

Simply using a bigger block size, like 10MB should help work the problem around while we study the way to take maximum advantage of the available resources.

Offline

#14 2020-12-27 18:26:24

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

Well, this is just what we suspected. There is some hardcoded, non-described limit in regards to the amount of memory a binary can use in the ESXi shell. Our latest improvements used more in-memory indexes to reduce the number of CPU cycles required to locate some block. This has caused that in the case of repositories exceeding 1,5TB, that limit is reached and a segfault is raised.

Older versions: 1.4.0.0 and below, didn't suffer from this issue as the kind of indexing they used was much more simple (only leading digit), that's why you can still use those versions without experimienting any problem.

We are already working in a solution that will allow to use fast indexing plus repositories hosting many tenths of terabytes, we will do that by optimizing the use of memory.

Please note, that a 1.5TB deduplicated repository can still host many tens of terabytes of real data.

Offline

#15 2021-01-01 20:01:18

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

We have solved this limit while still keeping the speed of an all in RAM approach. We are performing tests now with VMs well above 2TB and everything runs as smooth as with a small repository. Each new test requires some time, therefore, we will be able to release 1.4.3.0 in a couple of days.

Returning to a basic one significant digit index in RAM would make no sense, as speed would become slower as you add new data in big repositories > 1 TB

This new approach can be extended to even bigger repositories in future versions. Using a dedicated SSD to install (c)XSIBackup-DC for those of you willing to backup big (> 1TB) virtual machines is a very recommendable option, as we have moved the temporal data to the <install root>/tmp dir and some significant amount of random reads take place in that temp dir.

By adding --verbosity=10 to a backup job, you have a new counter, both per block and per job, which is the average block seek time. A fast M.2 SSD can reduce it, nonetheless in slower HDs, the new indexing system introduced in 1.4.3.0 takes advantage of the HD's cache. Using fast SSDs would increase differential backup times by some 3% at most.

Offline

#16 2021-01-05 13:12:28

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

(c)XSIBackup-DC 1.4.3.0 RC1 is ready for those of you willing to use it.
It is fully functional during 10 days, so you can try it at any server.

Offline

#17 2021-01-11 13:50:02

alexm
Member
Registered: 2018-04-22
Posts: 10

Re: VMWare ESXi: hardware Segmentation fault

I am getting a segmentation fault too. I'm running version 1.4.2.7. It was always 2 or 3 weeks since working fine and then segmentation fault.


xsibackup.log using --block-size=1M

-----------------------------------------------------------------------------------------------------------
--rotate option was detected, retrieving backups to prune...
-----------------------------------------------------------------------------------------------------------
ROTATION: set to 28 days
-----------------------------------------------------------------------------------------------------------
No backups to prune per day offset
-----------------------------------------------------------------------------------------------------------
Removed <tmp> dir OK
-----------------------------------------------------------------------------------------------------------
Unlocked backup OK
-----------------------------------------------------------------------------------------------------------
Segmentation fault


Then when I increased the block size to --block-size=20M it worked (from the GUI) and then it failed the first auto backup due to this:

error.log

2021-01-11T02:15:07 | Error code 131 at file signal.c, line 131 | Error description: some error was trapped SIGTERM (11) (11) while executing the job, error: No space left on device

xsibackup.log

Item number 2 in this job
-----------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------
2021-01-11T02:15:07 | Error code 131 at file signal.c, line 131 | Error description: some error was trapped SIGTERM (11) (11) while executing the job, error: No space left on device
-----------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------
SIGTERM (11) condition was trapped: check logs for more details

Which is weird because I have loads of spare space! So I tried running it manually again (I didn't touch anything else) using the GUI and it worked fine! The backup job is 5 VMs, 400GB.

Alex

Offline

#18 2021-01-11 17:47:58

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

Well, it doesn't necessarily mean that you run out of space in the target volume, there are other volumes implied, like temp file systems or log space.

In any case if your repo grew over 1.5 TB you should upgrade to 1.4.3.0 that will fix it, if the problem was due to lack of resources.

Offline

#19 2021-01-11 19:23:30

alexm
Member
Registered: 2018-04-22
Posts: 10

Re: VMWare ESXi: hardware Segmentation fault

you should upgrade to 1.4.3.0 that will fix it, if the problem was due to lack of resources

Thank you. I'll let you know how I get on.

Alex

Offline

#20 2021-01-18 09:33:43

alexm
Member
Registered: 2018-04-22
Posts: 10

Re: VMWare ESXi: hardware Segmentation fault

Right, I upgraded to version 1.4.3.0 and ran it manually from the GUI just to check and it ran fine. Then last night was the first time it ran automatically and it gave a segmentation fault:

Its running ESXi 7.0 Update 1

xsibackup.log

Performing a hot backup as per the --backup-how argument.
-----------------------------------------------------------------------------------------------------------
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.4.3.0: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 7 Major 0 Minor 1 Patch 0
-----------------------------------------------------------------------------------------------------------
License: 000906EA0000000000000000b47af1386b3c | (c)XSIBackup-DC
-----------------------------------------------------------------------------------------------------------
PID: 1548855, Running job as: root
-----------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-----------------------------------------------------------------------------------------------------------
Block size is 20.00 MB (20971520 bytes)
-----------------------------------------------------------------------------------------------------------
Performing --backup action
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC setting repository at /vmfs/volumes/QNAP_Backup/VMware
-----------------------------------------------------------------------------------------------------------
Item number 1 in this job
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: DC
-----------------------------------------------------------------------------------------------------------
(c)ESXi 7.0 or higher detected, all snapshots were removed for VM DC(1)
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : DC (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
Virtual Machine: DC
-----------------------------------------------------------------------------------------------------------
Backup start date: 2021-01-18T02:00:13
-----------------------------------------------------------------------------------------------------------
2021-01-18 02:00:13 | Backing up 25 files, total size is 64.42 GB
-----------------------------------------------------------------------------------------------------------
    NUMBER                                                         FILE             SIZE          PROGRESS
-----------------------------------------------------------------------------------------------------------
    1/25                                       vmx-dc-1679671574-1.vswp         86.00 MB    | Done   0.00%
-----------------------------------------------------------------------------------------------------------
    2/25                                                  vmware-25.log        203.51 KB    | Done   0.13%
-----------------------------------------------------------------------------------------------------------
    3/25                                                     vmware.log        231.41 KB    | Done   0.13%
-----------------------------------------------------------------------------------------------------------
    4/25                                                  vmware-30.log        244.23 KB    | Done   0.13%
-----------------------------------------------------------------------------------------------------------
    5/25                                                  vmware-29.log        149.74 KB    | Done   0.13%
-----------------------------------------------------------------------------------------------------------
    6/25                                                  vmware-28.log        212.69 KB    | Done   0.13%
-----------------------------------------------------------------------------------------------------------
    7/25                                                  vmware-27.log        285.46 KB    | Done   0.13%
-----------------------------------------------------------------------------------------------------------
    8/25                                                        dc.vmsd        486.00 B     | Done   0.13%
-----------------------------------------------------------------------------------------------------------
    9/25                                                        dc.vmxf          3.61 KB    | Done   0.13%
-----------------------------------------------------------------------------------------------------------
   10/25                                                       dc.nvram        328.50 KB    | Done   0.13%
-----------------------------------------------------------------------------------------------------------
   11/25                                                   dc-flat.vmdk         60.00 GB    | Done   0.13%
-----------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------
   12/25                                                        dc.vmdk        545.00 B     | Done  93.26%
-----------------------------------------------------------------------------------------------------------
   13/25                                                         dc.vmx          3.25 KB    | Done  93.26%
-----------------------------------------------------------------------------------------------------------
   14/25                                                  vmware-26.log        218.45 KB    | Done  93.26%
-----------------------------------------------------------------------------------------------------------
   15/25         vmx-dc-b95a0828c2a4a7a2dfb1d84155ec7718b0d2846e-1.vswp                    [open excluded]
-----------------------------------------------------------------------------------------------------------
   16/25                                                     dc.vmx.lck                 [skipped excluded]
-----------------------------------------------------------------------------------------------------------
   17/25                                                        dc.vmx~          3.24 KB    | Done  93.26%
-----------------------------------------------------------------------------------------------------------
   18/25                                               dc-39890d22.vswp                    [open excluded]
-----------------------------------------------------------------------------------------------------------
   19/25                                                    dc.vmsd.tmp         45.00 B     | Done  93.26%
-----------------------------------------------------------------------------------------------------------
   20/25                                                     dc.vmx.tmp          3.24 KB    | Done  93.26%
-----------------------------------------------------------------------------------------------------------
   21/25                                                     dc-aux.xml        123.00 B     | Done  93.26%
-----------------------------------------------------------------------------------------------------------
   22/25                                            dc-Snapshot149.vmsn        351.55 KB    | Done  93.26%
-----------------------------------------------------------------------------------------------------------
   23/25                                        dc-000001-sesparse.vmdk                 [skipped excluded]
-----------------------------------------------------------------------------------------------------------
   24/25                                                 dc-000001.vmdk                 [skipped excluded]
-----------------------------------------------------------------------------------------------------------
   25/25                                        dc-vss_manifests149.zip        314.00 B     | Done  93.27%Segmentation fault

error.log was empty.

Backup job was:

/scratch/XSI/XSIBackup-DC/xsibackup \
--backup \
"VMs(DC,XenforoDev,DNS1,WHM,DNS2)" \
"/vmfs/volumes/QNAP_Backup/VMware" \
--backup-how="hot" \
--rotate="28d" \
--mail-to="xxxxx@xxxxxx" \
--use-smtp="001" \
--description="Weekly VMware backup" \
--quiesce \
--block-size="20M" \
>> /scratch/XSI/XSIBackup-DC/var/log/xsibackup.log 2>&1

So I went into the GUI and ran it manually again and it worked fine!?

Alex

Last edited by alexm (2021-01-18 09:49:42)

Offline

#21 2021-01-18 14:49:47

alexm
Member
Registered: 2018-04-22
Posts: 10

Re: VMWare ESXi: hardware Segmentation fault

Ok, what I tried is --block-size="10M" but using a fresh folder and run automatically only... and it worked!

So I set up --rotate="2" and a cron that backed up every hour for 3 hours and it backed up and pruned the backup fine. big_smile

I'll try 20M again, but this time I'll try doing it in a fresh folder and run automatically only...

Offline

#22 2021-01-19 09:37:49

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

We haven't yet fully tested ESXi 7.0.0 U1 nor have we announced compatibility with it.

We hope that you are just using it for testing, as using any latest version of anything in production is equivalent, as we never lose the chance to say, to being the first to cross on thin ice. If you like risky sports, it's O.K., if on the contrary you want your job to be easier and sleep well at night, we would recommend that you use well tested software in production.

That said, and pending some more thorough tests on ESXi 7.0.0 U1, there isn't any fundamental reason why you shouldn't be able to run your backups.

Are you sure you haven't run our of space on the target volume? (c)XSIBackup-DC does not raise segfaults capriciously, it if did, then there must be some fundamental reason going on.

Offline

#23 2021-01-19 12:37:23

alexm
Member
Registered: 2018-04-22
Posts: 10

Re: VMWare ESXi: hardware Segmentation fault

I'm only running this on my home network. wink

Here's what I did so far:

  • Cleared /tmp and /scratch/XSI/XSIBackup-DC/tmp

  • Used fresh folders and my original folder

  • Tried automatic and manual GUI backups and a combination of both

  • Tried both 10M and 20M block size

It has passed all the tests using 1.4.3.0!

So either something screwed up originally not clearing /tmp? Or something is happening at that time that is causing it. The only thing I've found is a cPanel host on a different network is backing up at a similar time (you don't get the exact time, but I'm confident there is an overlap). Is there anything that would cause a segmentation fault on a slow speed disk? It's a QNAP NAS, connected by NFS to ESXi.

There's loads of space on the target volume!

Alex

Last edited by alexm (2021-01-19 12:54:59)

Offline

#24 2021-01-20 11:46:58

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: VMWare ESXi: hardware Segmentation fault

Thank you very much for your feedback.
We will revise v. 1.4.3.1 and reach you back ASAP.

Update 2021-01-20 17:46 UTC:

There isn't anything drastically new from 1.4.3.0 to 1.4.3.1, but some little amendments and the automatic update of the server side binary in over IP operations.

Overlapping jobs could only trigger an error in the event that a block was being accessed at the same time, which is highly unlikely to happen. This is in turn prevented by a file lock mechanism.

Server side block processing is performed by spare subprocesses. We have tested this to some extent with no issue.

Slow disks would not cause any issue, just as long as they are O.K.

Do pay attention to the possibility that the NFS mount is not well mounted. Your last error does really look like if you were writing to the mount folder and not to the mounted volume, we see this very often.

We have to add some logic to check for real available space in the target folder. It seems too obvious to miss, but the above commented situation is extremely frequent.

Also, if your link or disk is not very reliable, do use Sync NFS instead of Async NFS.

Offline

Board footer