Backups fail with Error code 1223 or 4574

Int · 2021-08-16 07:46:49

Hi,

on this backup job

"/scratch/XSI/XSIBackup-DC/xsibackup" \
--backup \
"VMs(^(IGMS|VPN|XBH))" \
/vmfs/volumes/backup/repo-$(date +%Y) \
--backup-how=hot \
--exclude="vmware-[0-9]{1,6}.log" \
--quiesce \
--timestamp=$(date +%Y%m%d000000) \
--use-smtp=1 \
--mail-to=dummy@dummy.com
>> /scratch/XSI/XSIBackup-DC/var/log/xsibackup.log 2>&1

the backup of three out of 19 VMs fails with these error codes (from the status Email):

· IGMS04: 2021-08-14T05:14:45 | Error code 1223 at file esxi.c, line 1223 | Error description: could not create snapshot in IGMS04, error: KO
· IGMS02: 2021-08-14T05:15:06 | Error code 1223 at file esxi.c, line 1223 | Error description: could not create snapshot in IGMS02, error: KO
· IGMS03: 2021-08-14T10:37:31 | Error code 4574 at file xsibackup.c, line 4574 | Error description: some error/s were raised while backing up: VMs(^(IGMS|VPN|XBH)), error count is: 2

Unfortunately I don't have more details about the errors because the log file /scratch/XSI/XSIBackup-DC/var/log/xsibackup.log
is empty. Btw. do you have an idea why the console output redirection in the job is not working?

The failing VMs have in common that they are all Windows machines (Windows 10 + Server 2016) and all have MS SQL Server running on them. The other non-failing VMs either don't have MS SQL server or are Linux machines.

Could it be that using option "--quiesce" doesn't play well with MS SQL Server?
I would appreciate any hints to solve this.

Finally a minor cosmetic flaw I saw. The email says as summary "error count is: 2 " but in fact the count of errors is 3

The ESXi server is Version 6.7.0 Update 3, XSIBackup-DC Datacenter is version 1.5.0.13

Best regards

Last edited by Int (2021-08-16 07:56:23)

admin · 2021-08-16 14:30:01

The error has little to do with (c)XSIBackup. It requested (c)ESXi to snapshot the guests OSs and they replied with an error.

Quiescing an OS that is running a DB server, especially if it is MS SQL server on Windows is not a trivial task. You need to properly configure all required services in the GUEST OS to be able to achieve that goal.

Make sure that you have (c)VMWare Tools installed, then inspect the Windows Event Viewer in search of the exact error that was raised by Windows when (c)ESXi requested the snapshot to be taken. It is most probably related to some auxiliary service not being active or installed.

You can start here for an aoverview of the subject: Quiescing Windows OSs

Int · 2021-08-16 15:42:47

It seems that I accidentally posted in the wrong forum - I found out now that the different forums are hard to distinguish because they all are titled "Index» General matters"

If you would have read my post more carefully you would have recognized that I talk about "XSIBackup-DC" several times.

XSIBackup-DC Datacenter is version 1.5.0.13

Could you please move this post to the correct forum and read it again taking into account that it is about "XSIBackup-DC"?

Last edited by Int (2021-08-16 15:43:32)

admin · 2021-08-16 18:35:37

The heading of our answer was a generic notice on Classic versions. Still, the same applies to DC. (c)XSIBackup does not produce the snapshot on its own, it is indeed the snapshot subsystem of (c)ESXi that does.

This is a common question regarding quiescing DB systems. We have prepared a specific post on that matter with links to resources. You do need to configure your Windows guests to allow quiescing the DB server.

Quiescing Windows OSs

As stated, quiescing a database server is not something you can achieve by just appending --quiesce to the job, it requires that the guest is previously prepared to be able to complete the quiescing flawlesly.

Each version of SQL Server may differ in the way quiescing is achieved, although it will generally imply installing and configuring general VSS services and other especific to SQL Server VSS.

Int · 2021-08-17 05:52:59

Thank you for clarifying in detail what the problems with quiescing VMs with MS SQL server are.
I will go through your post and see if I can get it working.

I read the chapter about "--quiesce" in the manual
https://33hops.com/xsibackup-dc-full-ma … ml#quiesce
but these possible problems were not mentioned there.

Perhaps you can link to your post "Quiescing Windows OSs" from the manual?
This would help others to find this information and not to run into the same trap as I did, I think.

Int · 2021-08-17 06:15:22

Can you tell something about these two minor questions I asked?

do you have an idea why the console output redirection in the job is not working?

/scratch/XSI/XSIBackup-DC/var/log/xsibackup.log is always empty.

Finally a minor cosmetic flaw I saw. The email says as summary "error count is: 2 " but in fact the count of errors is 3

admin · 2021-08-17 08:51:44

We will do so.

You have a missing "\" in the line before the last in your job.
We'll check the error count issue, it most probably has to do with the count base.

Int · 2021-08-17 09:46:18

Many thanks for having a second look at my job - I didn't spot that missing "\"

Int · 2021-08-23 08:07:47

Regarding the snapshot failures I found out the following:

For the machines that failed with Error code 1223 the vmware.log showed these error messages:

2021-08-14T05:15:01.439Z| vcpu-0| I125: ToolsBackup: changing quiesce state: COMMITTED -> FINISHING
2021-08-14T05:15:02.446Z| vcpu-0| I125: Msg_Post: Warning
2021-08-14T05:15:02.446Z| vcpu-0| I125: [msg.snapshot.quiesce.vmerr] The guest OS has reported an error during quiescing.
2021-08-14T05:15:02.446Z| vcpu-0| I125+ The error code was: 3
2021-08-14T05:15:02.446Z| vcpu-0| I125+ The error message was: Quiesce operation aborted
2021-08-14T05:15:02.446Z| vcpu-0| I125: ----------------------------------------
2021-08-14T05:15:02.448Z| vcpu-0| I125: ToolsBackup: changing quiesce state: FINISHING -> ERROR_WAIT

Which makes clear that the issue is within the VMs Windows OS.

Further investigation in the Windows event log showed multiple occurrences of Event ID 8 for Volsnap (a timeout occurred when Windows tried to make a snapshot).
But I could not make out a cause why this happened. There was sufficient free space to create the VSS snapshot and the I/O load was also low during that time.

For the machine that failed with Error code 4574 neither the vmware.log nor the Windows event log showed any helpful messages.

For all machines the "final" solution was to reboot the Windows OS. After the reboot the snapshot was working fine on all of them. Not a satisfying solution but at least it works now ...

I hope this information is helpful to others with the same issues.

admin · 2021-08-23 12:28:18

Thank you for your feedback, we are are sure It will be helpful to others.
What you have described is the tipicall issue regarding Windows quiescing.
Most probably some service was hanged and the reboot fixed It.

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

#1 2021-08-16 07:46:49

Backups fail with Error code 1223 or 4574

#2 2021-08-16 14:30:01

Re: Backups fail with Error code 1223 or 4574

#3 2021-08-16 15:42:47

Re: Backups fail with Error code 1223 or 4574

#4 2021-08-16 18:35:37

Re: Backups fail with Error code 1223 or 4574

#5 2021-08-17 05:52:59

Re: Backups fail with Error code 1223 or 4574

#6 2021-08-17 06:15:22

Re: Backups fail with Error code 1223 or 4574

#7 2021-08-17 08:51:44

Re: Backups fail with Error code 1223 or 4574

#8 2021-08-17 09:46:18

Re: Backups fail with Error code 1223 or 4574

#9 2021-08-23 08:07:47

Re: Backups fail with Error code 1223 or 4574

#10 2021-08-23 12:28:18

Re: Backups fail with Error code 1223 or 4574

Board footer