Error getting hash

mkl · 2017-12-02 15:25:20

Hi, I'm testing XSI-BackUp-Pro with XSITools for the deduplication of data.
Launching for example
"./xsibackup --backup-prog=xsitools --backup-point="/vmfs/volumes/BackUpVM/$(date +%Y%m)" --backup-type=custom --backup-vms=VM1
I always get (for all VM I Tested) the error:
"Cannot process block1, error getting hash, will retry. Warning:"
and it loops the message.

What's wrong?

Thanks

admin · 2017-12-02 17:13:09

Always use compression, set [b]--backup-prog=xsitools:z[/b] instead of [b]--backup-prog=xsitools[/b], LZO compression is very fast, so your backups won't take longer and the repositories will be reduced in size 50% aproximately.

Please post your output and your backup job if you want a more precise answer, otherwise we are just guessing and making asumptions.
You must have some previous errors that you didn't see. That error mesaage is thrown because XSITools can't access the disk to backup, there are a lot of different reasons why this could happen, like not being able to snapshot the VM, some other process holding the file, etc...

mkl · 2017-12-04 08:24:09

Thanks for the answer. I've yet tested with "z" compression but nothing.
However this is the output of my backup job

mkl · 2017-12-04 08:26:03

--backup-point created at /vmfs/volumes/BackUpVM/201712
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
WARNING: the /vmfs/volumes/BackUpVM/201712 dir is not an XSITOOLS repo, it will be initialized
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
E-mail Warnings
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The e-mail report will not be sent because of the followig reasons:
The --mail-from string has not been set
The --smtp-srv string has not been set
The --smtp-port string has not been set
The --smtp-usr string has not been set, you need --smtp-usr if --smtp-auth is other than -none-
The --smtp-pwd string has not been set, you need --smtp-pwd if --smtp-auth is other than -none-
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

mkl · 2017-12-04 08:27:10

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test-000001.vmdk
320 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)

sent 418 bytes received 35 bytes 906.00 bytes/sec
total size is 320 speedup is 0.71
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test-000002.vmdk
320 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)

sent 418 bytes received 35 bytes 302.00 bytes/sec
total size is 320 speedup is 0.71

mkl · 2017-12-04 08:28:12

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test-000002-delta.vmdk
17,190,912 100% 23.08MB/s 0:00:00 (xfr#1, to-chk=0/1)

sent 17,195,205 bytes received 35 bytes 34,390,480.00 bytes/sec
total size is 17,190,912 speedup is 1.00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test-Snapshot431.vmsn
30,813 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)

sent 30,916 bytes received 35 bytes 20,634.00 bytes/sec
total size is 30,813 speedup is 1.00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test.vmdk
546 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)

sent 637 bytes received 35 bytes 1,344.00 bytes/sec
total size is 546 speedup is 0.81
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Blocksize is 52428800 bytes
Compression has been set
Backing up via XSITools
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
File name: Win-Test-flat.vmdk, File size: 214748364800
Block size: 52428800, Block count: 4096
Disk usage: 44400902144
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Cannot process block 1, error getting hash, will retry. Warning:
Retrying...

admin · 2017-12-04 19:31:23

Please, run this command from the VM folder once the backup job gets to the error above and paste the output here.

dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 | openssl sha1

mkl · 2017-12-04 20:09:01

Ok thanks, this is the command and the output

/vmfs/volumes/51236600-21002fd4-6799-2c768a4e9360/Win-Test # dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 | openssl sha1
50+0 records in
50+0 records out
cceb46c84621368096e1a62bdfbb22898b1588e2

admin · 2017-12-04 20:49:54

Well, that is the expected output. Can you provide your exact version and build number by running this

vmware -v

Also, please run this extended command

dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1 2>/dev/null | grep "(stdin)=" | awk -F '=' '{print $2}' | sed -e 's/^ *//g' -e 's/ *$//g'

And post the output again

mkl · 2017-12-05 07:37:28

The version is VMware ESXi 5.1.0 build-799733

For the extended command I have no output

admin · 2017-12-05 08:11:32

We'll run new tests with this version and build, in the meanwhile you can just edit the [b]xsitools[/b] module file inside the [b]src[/b] folder

vi src/xsitools

from within the xsi-dir folder.
Go to line 16 and change the following

Change this

DDBIN="dd"
#DDBIN="$PWD/bin/dd"

to this

#DDBIN="dd"
DDBIN="$PWD/bin/dd"

This will tell (c) XSITools to use the provided [b]dd[/b] executable, inside the [b]bin[/b] folder, instead of the [b]Busybox[/b] binary bundled with ESXi.

Do this and try again

mkl · 2017-12-05 08:39:32

I've changed the string but the error with the backup is the same.

I've also relaunched the extended command specifyng the path of "dd"
"/vmfs/volumes/BackUpVM/xsi/bin/dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1 2>/dev/null | grep "(stdin)=" | awk -F '=' '{print $2}' | sed -e 's/^ *//g' -e 's/ *$//g'" and the output is still null

admin · 2017-12-05 08:50:24

Remove the 2>/dev/null in the long command and also the chained piped commands and see if you are able to grab the error.

dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1 2>/dev/null | grep "(stdin)=" | awk -F '=' '{print $2}' | sed -e 's/^ *//g' -e 's/ *$//g'

We haven't tested XSITools in ESXi 5.1 since some time ago, but all newer versions are tested.

There are subtle differences between command line utilities from one ESXi version to the next. Sometimes [b]grep[/b] or [b]sed[/b] arguments are changed and this causes some compounded commands to fail. We'll probably end up adding our own utilities to make sure they behave as expected for every build.

mkl · 2017-12-05 10:07:34

Removing "2>/dev/null" this is output:

50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.436811 seconds, 120 MB/s

So I analyze openssl and he version of vmware esx 5.1 the output doesn't contain "(stdin)=" at the begin of string.
So I change the string
"dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1 2>/dev/null | grep "(stdin)=" | awk -F '=' '{print $2}' | sed -e 's/^ *//g' -e 's/ *$//g'"
with
"dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1"
and now I have the hash

Changing xsitools script at row number 376
with
HASH=$( ( "$DDBIN" if="$FILEPATH" bs=$SUBBLKSIZ count=$SUBBLKCNT skip=$(( $(( ${icnt}-1 ))*${REGSUBBLKCNT} )) 2>/dev/null | "$OSBIN" sha1 ))

The string is also present in too rows. What is the fastest solution?

admin · 2017-12-05 12:32:31

The fastest solution for you would probably be to upgrade to 5.5 or later. We'll come up with a fix in short anyway. Thank you for taking the time to debug.

admin · 2018-07-26 19:20:37

Next Pro version 11.0.3 will contain a fix for this bug.

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

#1 2017-12-02 15:25:20

Error getting hash

#2 2017-12-02 17:13:09

Re: Error getting hash

#3 2017-12-04 08:24:09

Re: Error getting hash

#4 2017-12-04 08:26:03

Re: Error getting hash

#5 2017-12-04 08:27:10

Re: Error getting hash

#6 2017-12-04 08:28:12

Re: Error getting hash

#7 2017-12-04 19:31:23

Re: Error getting hash

#8 2017-12-04 20:09:01

Re: Error getting hash

#9 2017-12-04 20:49:54

Re: Error getting hash

#10 2017-12-05 07:37:28

Re: Error getting hash

#11 2017-12-05 08:11:32

Re: Error getting hash

#12 2017-12-05 08:39:32

Re: Error getting hash

#13 2017-12-05 08:50:24

Re: Error getting hash

#14 2017-12-05 10:07:34

Re: Error getting hash

#15 2017-12-05 12:32:31

Re: Error getting hash

#16 2018-07-26 19:20:37

Re: Error getting hash

Board footer