#1 2019-01-25 11:17:24

whitecrow
Member
Registered: 2018-10-11
Posts: 8

IP Backup to Synology - error: SHA1 chunk hashes check failed

Hello, I tested XSITools over IP - HOT backup to Synology with turn Certify checksum, I get this ERROR:

• [ Fri Jan 25 11:00:30 UTC 2019 ] ERROR (CLNCERT0), details [SRV] error: SHA1 chunk hashes check failed
• [ Fri Jan 25 10:52:05 UTC 2019 ] WARNING (XSITRM01), details Error 1 processing block number 183

• [ Fri Jan 25 10:59:34 UTC 2019 ] WARNING (XSITRM01), details Error 2 processing block number 315

Where is problem?
TNX

Offline

#2 2019-01-25 11:30:36

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

We don't state that XSITools will work on a Synology OS.

Offline

#3 2019-01-25 11:39:05

whitecrow
Member
Registered: 2018-10-11
Posts: 8

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

OK, then which tool is suitable for backup to Synology?

Last edited by whitecrow (2019-01-25 11:40:27)

Offline

#4 2019-01-25 16:33:26

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

That is a widely open question with no precise answer.

Mounting your Synology NAS as NFS3 or iSCSI is a trivial task, once you do it your volume will be available as a datastore and you'll be able to backup your VMs there via any of the XSIBackup --backup-prog programs, except those which are exclusively to be used over IP (Rsync).

Nevertheless, what you stated is "I tested XSITools over IP" which means that you directly tried to backup to your Synology NAS by connecting over IP to the SSH port, which as previously stated is not supported. XSIBackup may be used to backup over IP to other ESXi boxes (version and HW compatible) or to Linux servers, being our distro family of choice RHEL, Fedora, CentOS that is where we certify Linux features.

Offline

#5 2020-02-02 16:13:08

michib
Member
Registered: 2020-02-02
Posts: 4

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

whitecrow wrote:

Hello, I tested XSITools over IP - HOT backup to Synology with turn Certify checksum, I get this ERROR:

• [ Fri Jan 25 11:00:30 UTC 2019 ] ERROR (CLNCERT0), details [SRV] error: SHA1 chunk hashes check failed
• [ Fri Jan 25 10:52:05 UTC 2019 ] WARNING (XSITRM01), details Error 1 processing block number 183

• [ Fri Jan 25 10:59:34 UTC 2019 ] WARNING (XSITRM01), details Error 2 processing block number 315

Where is problem?
TNX

Hi whitecrow,

I see it is and old issue and different from mine...  but I am having pretty much the same while backupping with xsi over an nfs exported Synology raid1. Usually I did not enable sha1 checksum check but enabling it just fails with error  ERROR (CLNCERT0). I am wondering which Synology box were you using and/or if you managed to solve the problem you were having...

thanks, regards,

Offline

#6 2020-02-03 06:32:41

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

Read this first: https://33hops.com/xsibackup-disk-check … ption.html
Post your output, there's nothing we can say without knowing what you are trying to accomplish.

Offline

#7 2020-02-03 19:39:31

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

If you connect your ©Synology NAS via NFS or iSCSI, your remote FS is locally available to the ESXi server and backups are a mere local copy. Nonetheless, the fact that you are mounting over a network and via an intermediary protocol makes the setup more complex and thus more prone to errors, just because it comprises more elements than a plain local disk attached to a local controller.

Nevertheless, if everything is working fine you should expect backups to be made normally.

When you start to receive checksum mismatches, that is a sign that something is not working quite well. In case of locally attached disks, you could infere that some physical damage is causing the errors, you should then run some extensive SMART disgnostics to decide whether some of your HDs or SSDs needs to be replaced.

https://33hops.com/xsibackup-disk-check … ption.html

When it comes to NFS and iSCSI, the problem can as well be caused by a faulty disk, but it can also be due to some of the other elements playing a role in remotely mounting the storage volume, such as: cables, switches, NICs, etc...

Offline

#8 2020-02-05 19:29:53

michib
Member
Registered: 2020-02-02
Posts: 4

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

admin wrote:

If you connect your ©Synology NAS via NFS or iSCSI, your remote FS is locally available to the ESXi server and backups are a mere local copy. Nonetheless, the fact that you are mounting over a network and via an intermediary protocol makes the setup more complex and thus more prone to errors, just because it comprises more elements than a plain local disk attached to a local controller.

Nevertheless, if everything is working fine you should expect backups to be made normally.

When you start to receive checksum mismatches, that is a sign that something is not working quite well. In case of locally attached disks, you could infere that some physical damage is causing the errors, you should then run some extensive SMART disgnostics to decide whether some of your HDs or SSDs needs to be replaced.

https://33hops.com/xsibackup-disk-check … ption.html

When it comes to NFS and iSCSI, the problem can as well be caused by a faulty disk, but it can also be due to some of the other elements playing a role in remotely mounting the storage volume, such as: cables, switches, NICs, etc...

Hi admin,

Thanks for your reply. I post one XSI mail result down here. I was never used to certify backup but I decided it was a good idea when having to backup an administrative software with lots of data. It was then that checksum errors popped up. I added mail servers to that and also (see mail output) I receive sha checksum errors every time backup runs... sometimes the checksum is right (1 over 6). I checked smart my disk with extensive tests, then checked by changing raid1 volume (HGST vs. WD disks).. Now I opened a ticket to Synology and they make me try a memory test which seems to be ok. This is a pretty "enterprise" grade infrastructure: HP DL360 raid1+0 disks --> 10G fiber --> Cisco sg550xg -> 2x 1G bonding ethernet -> Synology (disaster recovery). I'll let you know what comes from Synology support.


Wed Feb 5 05:00:08 UTC 2020
XSIBACKUP-PRO 11.2.2.1 AT HOST:
esxi100.rifnet.local
IP v4: 151.1.254.146/ 255.255.255.0
VMware ESXi 6.0.0 build-5050593

Done hot backup (id: 10) using vmkfstools (no compression)
Backing up to folder /vmfs/volumes/archives/Mail/20200205050005
The backup room has been limited to 1000 Gb.
Available room in device /vmfs/volumes/archives/Mail before backup: 1000 Gb.
Sparse size on disk of the selected virtual machines: 277 Gb.
Needed room in device /vmfs/volumes/archives/Mail for backup: 236 Gb.
(Id)VM Name    State    Size (Gb)    Stop    Copy    Start    Time (min)    Speed (mb/s)
(8) linux-mail-55    ON    181/ 191    NO (hot backup)    sha1 CERTIFIED    -    92    33/ 35
(9) linux-mail-74    ON    55/ 86    NO (hot backup)    KO!    -    37    25/ 39
Last error raised for the above VM:
ERROR CLNCERT0, details: [linux-mail-74] error: sha1 hashes are different [linux-mail-74-flat.vmdk], details: 0
Available space in device /vmfs/volumes/archives/Mail after backup: 765 Gb.
Complete backup elapsed time: 130 min
The ESXi configuration was saved to /vmfs/volumes/archives/Mail/20200205050005

Offline

#9 2020-02-05 19:42:57

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

If you share some output, be it the full log, contact support not to risk sharing sensitive information. The error CLNCERT0 is quite narrowed down. As you are backing up to a local datastore, the problem is even simpler, as both checksums are calculated by the very same OpenSSL binary.

We bet your -flat.vmdk files are OK, you probably already tried them out.

Could be due to read errors, maybe some bad cable, some faulty memory module. The puzzling part is that they sometimes match, which almost excludes some software issue.

The bad side of certifying things via checksum is that while you don't you remain happily blind, while when you do, you get to the really nitty gritty basement of how things work. If error correction wasn't there computers wouldn't even work.

Offline

#10 2020-02-06 09:45:06

michib
Member
Registered: 2020-02-02
Posts: 4

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

admin wrote:

The bad side of certifying things via checksum is that while you don't you remain happily blind, while when you do, you get to the really nitty gritty basement of how things work. If error correction wasn't there computers wouldn't even work.

Yes, that's definitely true! But when copying from internal DA disks (actually SSD raid 1+0) I am able to get  every time the same sha1... Maybe it's related to the statistical error probability on rotational hard disks... We brought HGST because of their reliability but... even 10^15 when copying GBs gives the possibility of having an error bit even if it should not happen with this frequency thou...Anyhow, I'll keep you informed on how goes on with Synology. Also I will swap utp cables from Synology to the switches to see if that matters.

thanks,
regards,

Offline

#11 2020-02-06 09:58:27

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

Then you can say that device is not working as expected. Read errors are probably more common on rotational devices (I really don't have any figures on mind now), but when you get some checksum right out of 5, you can say that device is broken without any further comparison.

It might be just a bad cable, some bad disk or something that can be easily fixed, but so many errors indicate a failing device. Keep an eye on the inner cables if any, sometimes you are in a hurry and you just plug whatever you can find in. If some cable is below the spec, things will still work, but you would experience things like that.

Offline

#12 2020-02-12 09:18:46

michib
Member
Registered: 2020-02-02
Posts: 4

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

admin wrote:

Then you can say that device is not working as expected. Read errors are probably more common on rotational devices (I really don't have any figures on mind now), but when you get some checksum right out of 5, you can say that device is broken without any further comparison.

It might be just a bad cable, some bad disk or something that can be easily fixed, but so many errors indicate a failing device. Keep an eye on the inner cables if any, sometimes you are in a hurry and you just plug whatever you can find in. If some cable is below the spec, things will still work, but you would experience things like that.

Dear,

I checked pretty much all "the line" and nothing was there. So I got back this time checking XSI logs also... I noticed a strange thing: sha1 fails on the smallest vmdk, 16GB, and never on the second which holds mail data and is definitely bigger >150GB. So I checked 3 months of e-mail reports and it never failed on the bigger one: it seems very strange to me.... Also the mail sent says in subject 'No errors detected in backup jobs' but then body says:

 "Last error raised for the above VM:
ERROR CLNCERT0, details: [linux-mail-55] error: sha1 hashes are different [linux-mail-55-flat.vmdk], details: 0". 

What Last error means? Any ideas? I have actually brought an update for XSI but didnt update on production: do you think it could help solving this?

thanks,
regards.

Last edited by michib (2020-02-12 09:30:11)

Offline

#13 2020-02-12 12:40:58

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: IP Backup to Synology - error: SHA1 chunk hashes check failed

Create a new disk and copy the data to the new one and see if that helps the issue.

Offline

Board footer