#1 2017-12-02 15:25:20

mkl
Member
Registered: 2017-12-02
Posts: 9

Error getting hash

Hi, I'm testing XSI-BackUp-Pro with XSITools for the deduplication of data.
Launching for example
"./xsibackup --backup-prog=xsitools --backup-point="/vmfs/volumes/BackUpVM/$(date +%Y%m)" --backup-type=custom --backup-vms=VM1
I always get (for all VM I Tested) the error:
"Cannot process block1, error getting hash, will retry. Warning:"
and it loops the message.

What's wrong?

Thanks

Offline

#2 2017-12-02 17:13:09

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: Error getting hash

Always use compression, set --backup-prog=xsitools:z instead of --backup-prog=xsitools, LZO compression is very fast, so your backups won't take longer and the repositories will be reduced in size 50% aproximately.

Please post your output and your backup job if you want a more precise answer, otherwise we are just guessing and making asumptions.
You must have some previous errors that you didn't see. That error mesaage is thrown because XSITools can't access the disk to backup, there are a lot of different reasons why this could happen, like not being able to snapshot the VM, some other process holding the file, etc...

Offline

#3 2017-12-04 08:24:09

mkl
Member
Registered: 2017-12-02
Posts: 9

Re: Error getting hash

Thanks for the answer. I've yet tested with "z" compression but nothing.
However this is the output of my backup job

Offline

#4 2017-12-04 08:26:03

mkl
Member
Registered: 2017-12-02
Posts: 9

Re: Error getting hash

--backup-point created at /vmfs/volumes/BackUpVM/201712
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
WARNING: the /vmfs/volumes/BackUpVM/201712 dir is not an XSITOOLS repo, it will be initialized
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
E-mail Warnings
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The e-mail report will not be sent because of the followig reasons:
The --mail-from string has not been set
The --smtp-srv string has not been set
The --smtp-port string has not been set
The --smtp-usr string has not been set, you need --smtp-usr if --smtp-auth is other than -none-
The --smtp-pwd string has not been set, you need --smtp-pwd if --smtp-auth is other than -none-
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Offline

#5 2017-12-04 08:27:10

mkl
Member
Registered: 2017-12-02
Posts: 9

Re: Error getting hash

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test-000001.vmdk
            320 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 418 bytes  received 35 bytes  906.00 bytes/sec
total size is 320  speedup is 0.71
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test-000002.vmdk
            320 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 418 bytes  received 35 bytes  302.00 bytes/sec
total size is 320  speedup is 0.71

Offline

#6 2017-12-04 08:28:12

mkl
Member
Registered: 2017-12-02
Posts: 9

Re: Error getting hash

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test-000002-delta.vmdk
     17,190,912 100%   23.08MB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 17,195,205 bytes  received 35 bytes  34,390,480.00 bytes/sec
total size is 17,190,912  speedup is 1.00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test-Snapshot431.vmsn
         30,813 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 30,916 bytes  received 35 bytes  20,634.00 bytes/sec
total size is 30,813  speedup is 1.00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sending incremental file list
Win-Test.vmdk
            546 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 637 bytes  received 35 bytes  1,344.00 bytes/sec
total size is 546  speedup is 0.81
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Blocksize is 52428800 bytes
Compression has been set
Backing up via XSITools
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
File name: Win-Test-flat.vmdk, File size: 214748364800
Block size: 52428800, Block count: 4096
Disk usage: 44400902144
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Cannot process block 1, error getting hash, will retry. Warning:
Retrying...

Offline

#7 2017-12-04 19:31:23

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: Error getting hash

Please, run this command from the VM folder once the backup job gets to the error above and paste the output here.

dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 | openssl sha1

Offline

#8 2017-12-04 20:09:01

mkl
Member
Registered: 2017-12-02
Posts: 9

Re: Error getting hash

Ok thanks, this is the command and the output

/vmfs/volumes/51236600-21002fd4-6799-2c768a4e9360/Win-Test # dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 | openssl sha1
50+0 records in
50+0 records out
cceb46c84621368096e1a62bdfbb22898b1588e2

Offline

#9 2017-12-04 20:49:54

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: Error getting hash

Well, that is the expected output. Can you provide your exact version and build number by running this

vmware -v

Also, please run this extended command

dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1 2>/dev/null | grep "(stdin)=" | awk -F '=' '{print $2}' | sed -e 's/^ *//g' -e 's/ *$//g'

And post the output again

Offline

#10 2017-12-05 07:37:28

mkl
Member
Registered: 2017-12-02
Posts: 9

Re: Error getting hash

The version is VMware ESXi 5.1.0 build-799733

For the extended command I have no output

Offline

#11 2017-12-05 08:11:32

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: Error getting hash

We'll run new tests with this version and build, in the meanwhile you can just edit the xsitools module file inside the src folder

vi src/xsitools

from within the xsi-dir folder.
Go to line 16 and change the following

Change this

DDBIN="dd"
#DDBIN="$PWD/bin/dd"

to this

#DDBIN="dd"
DDBIN="$PWD/bin/dd"

This will tell (c) XSITools to use the provided dd executable, inside the bin folder, instead of the Busybox binary bundled with ESXi.

Do this and try again

Offline

#12 2017-12-05 08:39:32

mkl
Member
Registered: 2017-12-02
Posts: 9

Re: Error getting hash

I've changed the string but the error with the backup is the same.

I've also relaunched the extended command specifyng the path of "dd"
"/vmfs/volumes/BackUpVM/xsi/bin/dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1 2>/dev/null | grep "(stdin)=" | awk -F '=' '{print $2}' | sed -e 's/^ *//g' -e 's/ *$//g'" and the output is still null

Offline

#13 2017-12-05 08:50:24

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: Error getting hash

Remove the 2>/dev/null in the long command and also the chained piped commands and see if you are able to grab the error.

dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1 2>/dev/null | grep "(stdin)=" | awk -F '=' '{print $2}' | sed -e 's/^ *//g' -e 's/ *$//g'

We haven't tested XSITools in ESXi 5.1 since some time ago, but all newer versions are tested.

There are subtle differences between command line utilities from one ESXi version to the next. Sometimes grep or sed arguments are changed and this causes some compounded commands to fail. We'll probably end up adding our own utilities to make sure they behave as expected for every build.

Offline

#14 2017-12-05 10:07:34

mkl
Member
Registered: 2017-12-02
Posts: 9

Re: Error getting hash

Removing "2>/dev/null" this is output:

50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.436811 seconds, 120 MB/s

So I analyze openssl and he version of vmware esx 5.1 the output doesn't contain "(stdin)=" at the begin of string.
So I change the string
"dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1 2>/dev/null | grep "(stdin)=" | awk -F '=' '{print $2}' | sed -e 's/^ *//g' -e 's/ *$//g'"
with
"dd if="Win-Test-flat.vmdk" bs=1048576 count=50 skip=0 2>/dev/null | openssl sha1"
and now I have the hash

Changing xsitools script at row number 376
with
HASH=$( ( "$DDBIN" if="$FILEPATH" bs=$SUBBLKSIZ count=$SUBBLKCNT skip=$(( $(( ${icnt}-1 ))*${REGSUBBLKCNT} )) 2>/dev/null | "$OSBIN" sha1 ))

The string is also present in too rows. What is the fastest solution?

Offline

#15 2017-12-05 12:32:31

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: Error getting hash

The fastest solution for you would probably be to upgrade to 5.5 or later. We'll come up with a fix in short anyway. Thank you for taking the time to debug.

Offline

#16 2018-07-26 19:20:37

admin
Administrator
Registered: 2017-04-21
Posts: 1,370

Re: Error getting hash

Next Pro version 11.0.3 will contain a fix for this bug.

Offline

Board footer