#1 2017-06-12 16:07:26

sainsuper
Member
Registered: 2017-06-12
Posts: 17

error: first 50M mistmatch

Hello,
on my server i a have two virtual machine with old os (windows 2003 and windows 2000)
only on win200 machine i have this error:

• [ Mon Jun 12 13:29:27 UTC 2017 ] ERROR (DIFQMSH1) details [MagoServer] error: empty first 50M local [magoserver-flat.vmdk]
• [ Mon Jun 12 13:29:27 UTC 2017 ] ERROR (DIFQMSH3) details [MagoServer] error: first 50M mistmatch [magoserver-flat.vmdk]
• [ Mon Jun 12 13:29:28 UTC 2017 ] WARNING (DIFRMVMX) details [MagoServer] measure DIFRMVMX: took measure deleted remote .vmx file to reinitialize OneDiff


i use xsibackup pro 9.1.8
what i can do?
regard

Last edited by sainsuper (2017-06-12 16:32:18)

Offline

#2 2017-06-12 19:32:05

Daniel
Moderator
Registered: 2017-04-22
Posts: 60

Re: error: first 50M mistmatch

For some reason, the first 50M block in VM [MagoServer] is empty or set to zeros. We do not offer support for those ancient OSs, and neither does VMWare, so taking on account that their size has to be quite limited, just back them up fully, it'll probably take less than OneDiff. It does not make much sense to use OneDiff with this kind of OS.

In any case, the situation is a bit weird. Are those OSs running well?. From a .vmdk perspective the first 50M could be set to zero, but from the OS perspective it makes no sense, unless the level of fragmentation and block missplace is so high that left a hole at the very beggining of the physical disk, which is still very strange and unprobable.

Last edited by Daniel (2017-06-12 19:33:01)

Offline

#3 2017-06-13 07:42:59

sainsuper
Member
Registered: 2017-06-12
Posts: 17

Re: error: first 50M mistmatch

thanks for fast aswer, i'll made more test without onediff option.
thank you

Offline

#4 2017-08-31 09:44:45

jayce
Member
Registered: 2017-08-24
Posts: 12

Re: error: first 50M mistmatch

I have the exact same problem, but my OS is not *that* old... (Windows 2008 Server R2)

But i have an idea why first 50M blocks are empty...
Partitions Screenshot...

The first 300Mo on the disk are for the GPT/EFI partition...

Offline

#5 2017-08-31 11:02:47

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: error: first 50M mistmatch

Mmm, we'll review that check, it is very rare that the first 50M block is empty

https://en.wikipedia.org/wiki/GUID_Partition_Table

In any case, in the end it is up to VMWare and Microsoft to decide whether they allow empty space at the beginning of a physical disk. I'm not surprised that MS installer would generate such GPT partition, who else would?

We'll make thorough tests and will remove that check for MS OSs if, as it looks, they behave in such way.

Offline

#6 2017-08-31 13:54:16

jayce
Member
Registered: 2017-08-24
Posts: 12

Re: error: first 50M mistmatch

Ok, there must be another issue...
I tried to change the 50M test to 500M test. I wanted to circumvent this error waiting for next version of xsibackup...
So i changed the 50M by 500M in onediff script and tried...

And you know what ? I still have the error.... Checking further to see what really happens...

Offline

#7 2017-08-31 15:45:53

jayce
Member
Registered: 2017-08-24
Posts: 12

Re: error: first 50M mistmatch

Ok, so it looks like dd isn't working as expected :

[root@XServe3-ESXi:/xserve3/xsibackup] dd if="/vmfs/volumes/XServe3-500SSD-1/ZeusSrv/ZeusSrv-flat.vmdk" conv=notrunc  count=1 seek=0
???
   ????????U?1+0 records in
1+0 records out
[root@XServe3-ESXi:/xserve3/xsibackup] dd if="/vmfs/volumes/XServe3-500SSD-1/ZeusSrv/ZeusSrv-flat.vmdk" conv=notrunc count=1 seek=0 bs=50M
dd: /vmfs/volumes/XServe3-500SSD-1/ZeusSrv/ZeusSrv-flat.vmdk: Function not implemented
[root@XServe3-ESXi:/xserve3/xsibackup] dd if="/vmfs/volumes/XServe3-500SSD-1/ZeusSrv/ZeusSrv-flat.vmdk" conv=notrunc count=1 seek=0 bs=5000
dd: /vmfs/volumes/XServe3-500SSD-1/ZeusSrv/ZeusSrv-flat.vmdk: Invalid argument

Using bs argument in dd just make it go boooo.

Offline

#8 2017-09-01 06:43:58

jayce
Member
Registered: 2017-08-24
Posts: 12

Re: error: first 50M mistmatch

Hmmm... I checked on another server and i don't have this issue on the other server... They both are on same ESXI version...

Offline

#9 2017-09-01 10:15:04

sistemi
Member
Registered: 2017-08-29
Posts: 71

Re: error: first 50M mistmatch

During xsidiff testing i tried to backup a VM in "hostB" located on public hosting provider with 100Mbit internet bandwidth, launching the backup from "hostA" located in my office. The first try (or second i'm not sure) raise the 50M error, repeating the test (to verify the speed of xsidiff backup) the error didn't happen, with the backup successfully executed in a few minutes. Hope helps debugging...

Offline

#10 2017-09-01 10:32:28

jayce
Member
Registered: 2017-08-24
Posts: 12

Re: error: first 50M mistmatch

The first backup doesn't do the 50M test.
The second backup does it and may throw the error if dd is acting weird (or if the VM has empty block at beginning).
When throwing the error it remove the vmx file that cause the reset of the onediff backup.
So the 3rd backup doesn't do the 50M test because it does nearly a full backup like the first one.
4th backup will raise the error, etc.

Offline

#11 2017-09-01 11:25:14

sistemi
Member
Registered: 2017-08-29
Posts: 71

Re: error: first 50M mistmatch

Sorry, I mean the second diff backup, I saw that after the first error the next backup was "full", perhaps something was wrong during the remote rsync copy of the delta snaphot...

Offline

#12 2017-09-01 16:25:19

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: error: first 50M mistmatch

Jayce, I would not set a 500M check, as you will probably cause a "Cannot allocate memory" error, or at least you will multiply the chances that it happens.

This could be caused by a broken dd then. That would not surprise me at all, there are lots of bugs in different components of busybox in different ESXi versions.

We can fix this with a dd binary compiled for ESXi, but first I would like to locate the faulty dd, in case this is the problem. Can you all provide the ESXi version + build in which you have experienced this problem?.

UPDATE:

If a dd issue is confirmed we can provide a fix for those of you using the affected build number, it basically consists in a custom dd binary to be copied to the xsi-dir/bin directory plus a OneDiff module update.

Some older or more basic dd versions may not accept the 50M syntax and bs=52428800 should be used instead, we'll simplify our code to take just an integer byte count, just to prevent this hipotetical situation.

Offline

#13 2017-09-05 06:46:43

jayce
Member
Registered: 2017-08-24
Posts: 12

Re: error: first 50M mistmatch

You're right about the 500M, i juste wanted to try to have at least the full GPT partition to test if it was the issue.
I will do more tests today and give you details about builds. What annoys me is that on my 3 server i am supposed to have the same build number of ESXi and on 1 of them dd is working well.

Also this is not the M issue, i thought about that and tried even a simple bs=50 without more success.

Offline

#14 2017-09-05 09:16:18

jayce
Member
Registered: 2017-08-24
Posts: 12

Re: error: first 50M mistmatch

So, i identified why it worked on some and not on others.

If the VM is not in use, dd will work fine with the bs=50M parameter.
If the VM is in use, dd will throw the error, even if there is a snapshot and a delta file that should allow the use of dd.

My ESXi version is 6.0.0 Update 3 (Build 5572656).

Offline

#15 2017-09-05 20:20:58

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: error: first 50M mistmatch

Maybe some other process is holding the .vmdk file, running

lsof /path/to/your-flat.vmdk

Would offer information about the process accessing the file.
We are trying to replicate this issue, but will remove that check, or at least offer an argument to deactivate it if we can't find a permanent solution valid for all ESXi builds.
That 50M check is mainly useful with Linux disks, which will host the superblock in the first sectors. That check allows to know if the superblock has changed, and thus the contents of the disk. A positive check does not ensure disks are exact mirrors, but combined with the CID check offers a more than reasonable insight.

Offline

#16 2017-09-12 18:06:34

sistemi
Member
Registered: 2017-08-29
Posts: 71

Re: error: first 50M mistmatch

Happens randomly, vanishing the performance of OneDiff..., also still getting "Segmentation fault" and even "error seeking file" "file too large"

Offline

#17 2017-09-12 19:13:05

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: error: first 50M mistmatch

We are starting to mix different issues into this topic.
We had previously detected an issue with dd not being able to access a .vmdk file when checking its first 50M (Jayce pointed this out). This is most probably due to some other process holding it. We have not been able to reproduce this issue so far, and might not be a bug.

Getting a Segmentation fault or file too large from part of dd are traces of a different sort of problem.

dd uses C file handling functions, these functions generally use a type off_t to seek inside files. That error (file too large) means you have hit the limit of a data type. It could be due to your file being extremely big, or to a bug in the dd binary of your ESXi build.

We are willing to help you, but although we have insisted in getting some more details about this issue, we have nothing but comments.

If this topic has to be considered related to a real problem, please provide:

- Backup job string with all arguments (except passwords)
- ESXi version and build number.
- Logs containing the exact errors you are getting

Offline

#18 2017-09-19 13:49:51

noobstyle
Member
Registered: 2017-09-19
Posts: 8

Re: error: first 50M mistmatch

jayce wrote:

The first backup doesn't do the 50M test.
The second backup does it and may throw the error if dd is acting weird (or if the VM has empty block at beginning).
When throwing the error it remove the vmx file that cause the reset of the onediff backup.
So the 3rd backup doesn't do the 50M test because it does nearly a full backup like the first one.
4th backup will raise the error, etc.

Hello,
I have the same issue.

I have one question about it. When i upload a VM with OneDiff, (the first real full), the average upload speed is about 1.5mb/s. But when OneDiff backup full again cause the 50M error, the upload speed is about 40mb/s. So i guess it doesn't really upload a complete full backup for real. Is someone know how it works ? Don't really understand what's goin' on?

PS : when 50M error occurs, i have this too :
sed: bad regex '/vmfs/volumes/59a55e51-23ebeea2-f574-001b214be631/****ISO***/debian-8.9.0-i386-netinst.iso': Invalid preceding regular expression

Last edited by noobstyle (2017-09-19 13:53:42)

Offline

#19 2017-09-20 19:24:50

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: error: first 50M mistmatch

OneDiff stores the data between backups in a snapshot, and the subsequent backup cycle that diff data is sent to the backup mirror and integrated with the previous data. XSIBackup is a script, you have all the details in the source code itself, should you be curious to know more in depth.

What you are describing is probably an error and a derived error, which is caused by the first.
We really want to help you, but although this thread has been open for quite a long time, we still don't have any details.

If this topic has to be considered related to a real problem, please provide:

- Backup job string with all arguments (except passwords)
- ESXi version and build number.
- Logs containing the exact errors you are getting

Offline

#20 2017-09-21 14:00:19

noobstyle
Member
Registered: 2017-09-19
Posts: 8

Re: error: first 50M mistmatch

hello,

i'm on ESXI 5.1.0, 1483097

this is my cron :

/vmfs/volumes/datastore1/xsi-dir/xsibackup backupid=01 --time="Mon 02:00|Tue 15:45|Wed 13:43|Thu 15:06|Fri 02:00|Sat 02:00|Sun 02:00" \ --backup-prog=onediff --backup-point="x.x.x.x:22:/vmfs/volumes/datastore1" \ --backup-type=custom --backup-vms="debian" \ --on-success="backupid->02" --on-error="backupid->02"
/vmfs/volumes/datastore1/xsi-dir/xsibackup backupid=02 --host=x.x.x.x --backup-prog=xsitools:z \ --backup-point="/vmfs/volumes/datastore1/$(date '+%a')-xsitools" --backup-type=custom \ --backup-vms="debian" --override=xsibakfilter


this is the logs:

2017-09-20T13:49:37|  Syncronizing configuration files
---------------------------------------------------------------------------------------------------------------------------------
sed: bad regex '/vmfs/volumes/59a55e51-23ebeea2-f574-001b214be631/****ISO***/debian-8.9.0-i386-netinst.iso': Invalid preceding regular expression
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:49:42|  Removing hot snapshot & merging data
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:52:00|  [debian] info: OneDiff backup differential data (44) merged with base data
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:52:03|  [debian] info: diff mirror VM reloaded
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:52:03|  [debian] info: deleting snapshot at [debian_XSIBAK] Id 16...
2017-09-20T13:52:37|  [debian] info: all snapshots removed
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:52:37|  [debian] info: Updating CID at 192.168.238.1:22:/vmfs/volumes/datastore1/debian/debian.vmdk
2017-09-20T13:52:38|  [debian] info: finished OneDiff backup, CID updated
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:52:38|  [debian] info: performing quick check...
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:52:39|  [debian] info: file [debian-flat.vmdk]...
2017-09-20T13:52:40|  [debian] info: file size check | [0;32mOK[0m [ 17179869184B | 17179869184B ]
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:52:40|  [debian] info: comparing first 50M of [debian-flat.vmdk]...
2017-09-20T13:52:44|  [0;31m[debian] error DIFQMSH1: empty first 50M local (inexistent file?)[0m...
2017-09-20T13:52:44|  [0;31m[debian] error DIFQMSH3: first 50M mistmatch[0m...
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:52:46|  [0;36m[debian] measure DIFRMVMX: took measure deleted remote .vmx file to reinitialize OneDiff[0m
The ESXi configuration was saved to "x.x.x.x:22:/vmfs/volumes/datastore1"
---------------------------------------------------------------------------------------------------------------------------------
2017-09-20T13:53:06|  [0;31mErrors detected in backup, check logs[0m

Last edited by noobstyle (2017-09-21 14:02:49)

Offline

#21 2017-09-21 17:24:14

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: error: first 50M mistmatch

O.K., this is something for us to check. I guess you left the CD .iso file attached to the VM

/vmfs/volumes/59a55e51-23ebeea2-f574-001b214be631/****ISO***/debian-8.9.0-i386-netinst.iso

On the other side the asterisks in the path may be causing the sed regexp to fail.
Now I'm wondering if this could be causing side effects in regards to the first empty 50M.
Please, remove that .iso file and give feedback.

Offline

#22 2017-09-22 09:15:14

noobstyle
Member
Registered: 2017-09-19
Posts: 8

Re: error: first 50M mistmatch

You were right about regex error. But still have 50M problem.


First run

Hot backup selected for VM: [ubuntu], will not be switched off
---------------------------------------------------------------------------------------------------
[ubuntu] info: quick size check...
---------------------------------------------------------------------------------------------------
[ubuntu] info: file [ubuntu-flat.vmdk]...
[ubuntu] info: file size check | OK [ 17179869184 bytes | 17179869184 bytes ]
---------------------------------------------------------------------------------------------------
OneDiff
---------------------------------------------------------------------------------------------------
[ubuntu] info: OneDiff backup first run, removing snapshots
[ubuntu] info: all snapshots removed
---------------------------------------------------------------------------------------------------
Snapshot & Quiescing
---------------------------------------------------------------------------------------------------
[ubuntu] info: the VM will now be quiesced to ensure proper data handling
[ubuntu] info: snapshot taken, quiescing status: YES
---------------------------------------------------------------------------------------------------
[ubuntu] error OVMXCHK1: VMX file check 1 fail, details: first line in .vmx file is not [.encoding = "UTF-8"], has this file been manually modified/edited?
Syncronizing config files
---------------------------------------------------------------------------------------------------
Backing up virtual disks...
---------------------------------------------------------------------------------------------------
scsi0:0.fileName = "ubuntu.vmdk"
---------------------------------------------------------------------------------------------------
Destination disk format: VMFS thin-provisioned
Cloning disk '/vmfs/volumes/datastore1/ubuntu/ubuntu.vmdk'...
Clone: 100% done.
---------------------------------------------------------------------------------------------------
[ubuntu] info: backup complete
[ubuntu] info: finished doing a first copy in a diff scope, subsequent backups will be differential
[ubuntu] info: should you need to use the mirror VM, register it manually at the backup ESXi host
---------------------------------------------------------------------------------------------------
The ESXi configuration was saved to "/vmfs/volumes/backup"
---------------------------------------------------------------------------------------------------
Errors detected in backup, check logs
---------------------------------------------------------------------------------------------------
Backup finished
Tip: no chained backups scheduled, set --on-success and/or --on-error arguments to chain a backup
Killed

Offline

#23 2017-09-22 09:19:18

noobstyle
Member
Registered: 2017-09-19
Posts: 8

Re: error: first 50M mistmatch

summary of second run :

------------------------------------------------------------------------------------------------------------------------------------------------
OneDiff
-------------------------------------------------------------------------------------------------------------------------------------------------
Mirror VM exists and the system disks CID are the same, starting OneDiff...
[ubuntu] info: length of .vmx file before OneDiff phase is 2688
-------------------------------------------------------------------------------------------------------------------------------------------------
Initiating OneDiff differential backup...
-------------------------------------------------------------------------------------------------------------------------------------------------
Syncronizing configuration files
-------------------------------------------------------------------------------------------------------------------------------------------------
[ubuntu] info: length of remote .vmx file after copying it is 2688
[ubuntu] info: length of remote .vmx after replacing name is 2695
[ubuntu] info: name changed to [ubuntu_XSIBAK]
-------------------------------------------------------------------------------------------------------------------------------------------------
Removing hot snapshot & merging data
-------------------------------------------------------------------------------------------------------------------------------------------------
[ubuntu] info: OneDiff backup differential data (9) merged with base data
-------------------------------------------------------------------------------------------------------------------------------------------------
[ubuntu] info: diff mirror VM reloaded
-------------------------------------------------------------------------------------------------------------------------------------------------
[ubuntu] info: deleting snapshot at [ubuntu_XSIBAK] Id 9...
[ubuntu] info: all snapshots removed
-------------------------------------------------------------------------------------------------------------------------------------------------
[ubuntu] info: Updating CID at /vmfs/volumes/backup/ubuntu/ubuntu.vmdk
[ubuntu] info: finished OneDiff backup, CID updated
-------------------------------------------------------------------------------------------------------------------------------------------------
[ubuntu] info: comparing first 50M of [ubuntu-flat.vmdk]...
[ubuntu] error DIFQMSH1: empty first 50M local (inexistent file?)...
[ubuntu] error DIFQMSH2: empty first 50M remote (inexistent file?)...
[ubuntu] info: first 50M check | OK [  ]
-------------------------------------------------------------------------------------------------------------------------------------------------
[ubuntu] measure DIFRMVMX: took measure deleted remote .vmx file to reinitialize OneDiff
The ESXi configuration was saved to "/vmfs/volumes/backup"
-------------------------------------------------------------------------------------------------------------------------------------------------
Backup finished
Killed

Offline

#24 2017-09-25 08:07:02

jayce
Member
Registered: 2017-08-24
Posts: 12

Re: error: first 50M mistmatch

admin wrote:

Maybe some other process is holding the .vmdk file, running

lsof /path/to/your-flat.vmdk

Would offer information about the process accessing the file.
We are trying to replicate this issue, but will remove that check, or at least offer an argument to deactivate it if we can't find a permanent solution valid for all ESXi builds.
That 50M check is mainly useful with Linux disks, which will host the superblock in the first sectors. That check allows to know if the superblock has changed, and thus the contents of the disk. A positive check does not ensure disks are exact mirrors, but combined with the CID check offers a more than reasonable insight.

Sorry for the delay, i had some other urgent issues to manage... Here are the results of the lsof :

<code>[root@XServe3-ESXi:~] lsof |grep ZeusSrv-flat
281135      vmx                   FILE                       82   /vmfs/volumes/55785226-1e682e1a-0f8c-001e52f3e01c/ZeusSrv/ZeusSrv-flat.vmdk

[root@XServe3-ESXi:~] esxcli vm process list
ZeusSrv
   World ID: 281136
   Process ID: 0
   VMX Cartel ID: 281135
   UUID: 56 4d 66 f3 34 97 8a 64-d5 22 9f c1 89 bc 57 bb
   Display Name: ZeusSrv
   Config File: /vmfs/volumes/55785226-1e682e1a-0f8c-001e52f3e01c/ZeusSrv/ZeusSrv.vmx

[root@XServe3-ESXi:~] grep vmdk /vmfs/volumes/XServe3-500SSD-1/ZeusSrv/ZeusSrv.vmx
scsi0:0.fileName = "ZeusSrv-000002.vmdk"
</code>

Offline

#25 2017-09-25 08:29:55

sistemi
Member
Registered: 2017-08-29
Posts: 71

Re: error: first 50M mistmatch

admin wrote:

We are starting to mix different issues into this topic.
We had previously detected an issue with dd not being able to access a .vmdk file when checking its first 50M (Jayce pointed this out). This is most probably due to some other process holding it. We have not been able to reproduce this issue so far, and might not be a bug.

Getting a Segmentation fault or file too large from part of dd are traces of a different sort of problem.

dd uses C file handling functions, these functions generally use a type off_t to seek inside files. That error (file too large) means you have hit the limit of a data type. It could be due to your file being extremely big, or to a bug in the dd binary of your ESXi build.

We are willing to help you, but although we have insisted in getting some more details about this issue, we have nothing but comments.

If this topic has to be considered related to a real problem, please provide:

- Backup job string with all arguments (except passwords)
- ESXi version and build number.
- Logs containing the exact errors you are getting

Well, I was busy too, but during this time I set up a new ESXI host (6.0.0) redirecting all remote backups to this host, with internal hdd storage. The one that is having problems has a NAS (QNAP TS-420) attached in the local network.

This was about one week ago, and until yesterday there were no errors.

So the problem has moved to the NFS datastore created linking the NAS RAID volume, don't know if could be the NAS hardware, firmware, or something else, but i thing nothing related to XSIBackup itself.

Hope this helps investigating.

Offline

Board footer