#1 2017-10-24 11:04:57

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

error: first 50M mistmatch

Hi.

I’ve tested your script earlier and just recently I bought the PRO version because I find it a very cost-effective and easy way to back up my VMs. Kudos on a great product.

Now, however I’m having a problem but let me just briefly list my environment:

•    HP Proliant running ESXi 6, update 3
•    4 VMs (1 Windows Server 2008 R2 Core, 1 Windows Server 2008 R2 and two linux-flavoured servers)

I set up xsi and onedif and everything went smoothly. All was backed up and no problems were seen.

Looking through the logs I did notice the warning about quiescing being disabled due to Windows on MBR partitions and needless to say - I went about solving that. wink

Long story short:

•    I converted the disk to GPT (I can revive the link if needed)
•    Deleted the original VM from inventory
•    Recreated VM with UEFI and hardware level 11
•    Connected VM to converted disk
•    Booted the VM a couple of times
•    Everything came up smoothly (YAY!)

It is my AD-controller so I was very pleased when it all came back. I then proceeded to delete the XSIBAK snapshot (and clean out the target snapshot-folder for good meassure) before I manually started my xsi-job. And again - it worked like a carm. It did complain about not finding the .nvram and .nvxm file but looking in the folder they were both there and I have not followed up on this and I am unsure if this is related to my problem.

The problem arrises when I re-run the xsi-job the second time and it only presents itself when backing up the converted disk VM:

•    It starts
•    Detects the VM
•    Quiesces the VM succsessfully
•    Compares sizes and CIDs which checks out
•    Then it backs up the disks
•    When it does the final comparisson it fails with: "error DIFQMSH3: first 50M mistmatch..."

It then cleans up, removes the .xmx to initialize a full backup the next time and then kills itself.

I have attached the log from the second run and hopefully you can help me shed some light on this issue.

Appreciate your answer.

Best regards,

Ulf Thomas

Offline

#2 2017-10-24 11:06:54

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

Re: error: first 50M mistmatch

Excerts from the logfile. (I was not able to post the complete log due to a 404 error)

[nonet01] info: file [nonet01-flat.vmdk]...
[nonet01] info: file size check | OK [ 48318382080 bytes | 48318382080 bytes ]
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[nonet01] info: remote VMX file size 2565
System disk CIDs b0566e59|b0566e59
-------------------------------------------------------------------------------------------------------------------------------------------------------------
OneDiff
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Mirror VM exists and the system disks CID are the same, starting OneDiff...
[nonet01] info: length of .vmx file before OneDiff phase is 2564
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[nonet01] info: Updating CID at 192.168.1.11:22:/vmfs/volumes/RAIDBackup/snapshot/nonet01/nonet01.vmdk
[nonet01] info: finished OneDiff backup, CID updated
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[nonet01] info: performing quick check...
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[nonet01] info: file [nonet01-flat.vmdk]...
[nonet01] info: file size check | OK [ 48318382080B | 48318382080B ]
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[nonet01] info: comparing first 50M of [nonet01-flat.vmdk]...
[nonet01] error DIFQMSH3: first 50M mistmatch...
-------------------------------------------------------------------------------------------------------------------------------------------------------------

Offline

#3 2017-10-24 14:24:46

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: error: first 50M mistmatch

We have reprogrammed that check and now it has been expanded to the first 500M (which does not take much longer than 50M), you can download v 10.0.4 which was released this weekend and includes that improvement. We have not only expanded the length of data to check, but also improved that routine.

That mistmatch is, most probably, due to a problem accessing any of the two files in comparison. We'll continue to improve this check, but latest version will, at least, offer you more detail about what is really happening.

Make sure that you don't have any ghost xsibackup-rsync process hanging around. Execute: ps -c | grep xsi and check what you have in memory.

Offline

#4 2017-10-24 14:58:21

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

Re: error: first 50M mistmatch

That is good news indeed. I will download the updated version at once and report back. Thank you.

I do have a follow-up question to something I mentioned discovering: After cleaning out my snapshot folder and removing all snapshots xsi still complains that it cannot find the .vmxf and .nvram-files for one of the VMs:

[Sophos] info: created dir to host VM backup
[Sophos] info: VMX file copied succesfully
[Sophos] info: VMSD file copied succesfully
[Sophos] info: VMXF file copied succesfully.
[Sophos] info: NVRAM file copied succesfully.
[exch02] info: created dir to host VM backup
[exch02] info: VMX file copied succesfully
[exch02] info: VMSD file copied succesfully
[exch02] info: VMXF file copied succesfully.
[exch02] info: NVRAM file copied succesfully.
No .vmxf file found for nonet01 (32)
No .nvram file found for nonet01 (32)
[nonet01] info: created dir to host VM backup
[nonet01] info: VMX file copied succesfully
[nonet01] info: VMSD file copied succesfully

Offline

#5 2017-10-24 15:06:46

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: error: first 50M mistmatch

It's just a warning, you can ignore it.

Offline

#6 2017-10-24 15:08:23

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

Re: error: first 50M mistmatch

Hi.

Just redid the test with your latest version and I am sorry to say the results are the same:

[nonet01] info: comparing first 500M of [nonet01-flat.vmdk]...
[nonet01] error DIFQMSH3: first 500M mistmatch...

I am also seeing this in the Vsphere client when selecting the snapshot: (Also noticed this before trying your latest version)

Configuration issues

Virtual machine disks consolidation is needed

Last edited by ulfthomas (2017-10-24 15:11:13)

Offline

#7 2017-10-24 16:11:59

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

Re: error: first 50M mistmatch

Offline

#8 2017-10-24 16:36:17

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: error: first 50M mistmatch

It shouldn't, as the 500M check is performed once all snapshots are removed from the target VM and the ghost partition is created during the snapshot process. Can you confirm if the ghost partition is persistent after removing the snapshot?, check both the source and target VMs.

Offline

#9 2017-10-24 17:01:10

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

Re: error: first 50M mistmatch

admin wrote:

It shouldn't, as the 500M check is performed once all snapshots are removed from the target VM and the ghost partition is created during the snapshot process. Can you confirm if the ghost partition is persistent after removing the snapshot?, check both the source and target VMs.

Both the VM and its snapshot contains the following partitions:


Partition 1 System 100MB
Partition 4 Unknwon 256KB
Partition 2 Reserved 127MB
Partition 3 Primary 39GB

This is verified after taking the first Onediff backup (with all snapshots and folders empty) and with the VM in an offline state.

Offline

#10 2017-10-24 17:20:36

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

Re: error: first 50M mistmatch

Did another test:

- Cleaned out all the snapshots and folders
- Shutdown VM
- Started Onediff backup with --backup-type=warm
- Started the second Onediff backup with the VM still off and that passed the comparisson step
- Restarted the VM
- Started another Onediff backup and now it failed the comparisson step

Offline

#11 2017-10-24 18:03:56

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

Re: error: first 50M mistmatch

A couple of more tests:

- Cleaned out all the snaphosts and folders
- Initiated the first Onediff backup with additional switch --backup-how=warm: Backup successful
- Initiated the second Onediff backup with the same switch: Backup successful
- Initiated the third Onediff backup removing warm switch but adding --snapshot=dontquiesce: Backup successful
- Initiated the forth Onediff backup removing both warm and snapshot switch: failed

Wouldn't this indicate that it could be related to the quiescing of the VM?

Last edited by ulfthomas (2017-10-24 18:09:37)

Offline

#12 2017-10-25 07:14:53

sistemi
Member
Registered: 2017-08-29
Posts: 71

Re: error: first 50M mistmatch

ulfthomas wrote:

Long story short:

•    I converted the disk to GPT (I can revive the link if needed)

...

I'm interested at the link, can you provide it?

Offline

#13 2017-10-25 12:59:04

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: error: first 50M mistmatch

sistemi wrote:
ulfthomas wrote:

Long story short:

•    I converted the disk to GPT (I can revive the link if needed)

...

I'm interested at the link, can you provide it?

You have a link to a free tool in our Most Common Issues page here: https://33hops.com/xsibackup-most-common-issues.html

Offline

#14 2017-10-25 13:01:59

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: error: first 50M mistmatch

ulfthomas wrote:

A couple of more tests:

- Cleaned out all the snaphosts and folders
- Initiated the first Onediff backup with additional switch --backup-how=warm: Backup successful
- Initiated the second Onediff backup with the same switch: Backup successful
- Initiated the third Onediff backup removing warm switch but adding --snapshot=dontquiesce: Backup successful
- Initiated the forth Onediff backup removing both warm and snapshot switch: failed

Wouldn't this indicate that it could be related to the quiescing of the VM?

Yes, it could be related to the VMWare issue that you pointed out. Remove the 500M check with the options flag --options=no-500M-check and see if this helps the problem. Nevertheless, the fact that the resulting VM is not an exact copy of the original one when quiescing, is something to be concerned about. I guess VMWare should solve this issue.

We are working on this issue. We will offer a solution or workaround in short.

Offline

#15 2017-10-25 15:24:00

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

Re: error: first 50M mistmatch

admin wrote:
sistemi wrote:
ulfthomas wrote:

Long story short:

•    I converted the disk to GPT (I can revive the link if needed)

...

I'm interested at the link, can you provide it?

You have a link to a free tool in our Most Common Issues page here: https://33hops.com/xsibackup-most-common-issues.html

That is indeed the link I was using (I'm sorry but I forgot). smile There is a couple of typos in that article:

gptgen.exe -w \.physicaldrive0
bcdboot c:windows /s s: /f UEFI

Needs to be changed to:

gptgen.exe -w \\.\\physicaldrive0
bcdboot c:\windows /s s: /f UEFI


Else it will fail.

Last edited by ulfthomas (2017-10-25 15:25:58)

Offline

#16 2017-10-25 15:34:01

ulfthomas
Member
Registered: 2017-10-24
Posts: 10

Re: error: first 50M mistmatch

admin wrote:

Yes, it could be related to the VMWare issue that you pointed out. Remove the 500M check with the options flag --options=no-500M-check and see if this helps the problem. Nevertheless, the fact that the resulting VM is not an exact copy of the original one when quiescing, is something to be concerned about. I guess VMWare should solve this issue.

I fully agree. But Vmware do state in this article that this is by design and by that I understand the presence of this partition should not be of any concern. Are you saying that the disks are not identical or could it be an error that happens within the test itself? Could the disks be compared in any other ways?


We are working on this issue. We will offer a solution or workaround in short.

Highly appreciated. Let me know if you need any help testing or debugging. I'll assist with as much as I can.

Last edited by ulfthomas (2017-10-25 15:35:23)

Offline

#17 2017-10-26 14:36:28

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: error: first 50M mistmatch

ulfthomas wrote:
admin wrote:
sistemi wrote:

I'm interested at the link, can you provide it?

You have a link to a free tool in our Most Common Issues page here: https://33hops.com/xsibackup-most-common-issues.html

That is indeed the link I was using (I'm sorry but I forgot). smile There is a couple of typos in that article:

gptgen.exe -w \.physicaldrive0
bcdboot c:windows /s s: /f UEFI

Needs to be changed to:

gptgen.exe -w \\.\\physicaldrive0
bcdboot c:\windows /s s: /f UEFI


Else it will fail.

Thank you for the info, but you should communicate that to the software authors at: https://sabrnet.wzk.cz/2015/03/convert- … data-loss/
There is a textbox where you can leave some comments.

Offline

#18 2017-10-26 14:37:46

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: error: first 50M mistmatch

ulfthomas wrote:
admin wrote:

Yes, it could be related to the VMWare issue that you pointed out. Remove the 500M check with the options flag --options=no-500M-check and see if this helps the problem. Nevertheless, the fact that the resulting VM is not an exact copy of the original one when quiescing, is something to be concerned about. I guess VMWare should solve this issue.

I fully agree. But Vmware do state in this article that this is by design and by that I understand the presence of this partition should not be of any concern. Are you saying that the disks are not identical or could it be an error that happens within the test itself? Could the disks be compared in any other ways?


We are working on this issue. We will offer a solution or workaround in short.

Highly appreciated. Let me know if you need any help testing or debugging. I'll assist with as much as I can.

Thank you for your offer, we will get in contact with you once we have some new code to test. You can, by now, just turn off the check.

Offline

#19 2018-09-05 03:05:07

epicurean
Member
Registered: 2017-12-04
Posts: 20

Re: error: first 50M mistmatch

Is the --options=no-500M-check still valid for ver 11.02 and above?
Has this issue been resolved?

Offline

#20 2018-09-05 09:27:09

admin
Administrator
Registered: 2017-04-21
Posts: 1,362

Re: error: first 50M mistmatch

Yes, the option is still valid of course. You should accomodate the portion in bold characters to what you have in conf/xsitopts, in case you have tweaked the CHKMB variable.

--options=no-500M-check

The size and first 500M check AKA Trivial Check is extremely usefull 99% of the times. There are nevertheless some weird workarounds in VMWare snapshotting system related to Windows OSs, especially when quiescing them.

You cannot create a ghost partition on the fly, leave it there and state that is by design. Well, you can, but you are then renouncing to being taken seriously. We all know the by design and we have still not come to a conclussion are common out there.

Using a Linux OS within ESXi is something seamless and will work like a charm when quiescing the OS.

Windows OSs are something apart. It's obvious that this two monster companies are having an open commercial war, as Microsoft tries to force their users to use Hyper-V. There are bugs, especially in everything related to quiescing, that have plainly not been resolved. That is most probably due to MS putting efforts in making their OSs not fully compatible with VMWare.

It's not a coincidence that quiescing MS Operating Systems be the battle ground, as this operation is critical to properly backing up mission critical servers, like Exchange or SQL Server, which are some of the MS flagships.

First of all, if you want to run mission critical services, use Unix or Linux. If you already have MS in place and still need to back them up, follow our guide to quiescing Microsoft OSs Troubleshooting Windows Snapshots In Esxi

Be prepared to still receive some quiescing errors from time to time. You can use the --options=no-500M-check argument at your own risk, but take the following on account:

If you take your time to quiesce some Microsoft Operating System in a mission critical 24x7 service, you might still find yourself fighting in the mud against some invisible bugs that cause quiescing operations to time out or raise errors in the MS Event Viewer, which are in turn catalogued as "ignorable" by the Microsoft Knowledge base articles.

Thus, it's probably wiser to use one of two approaches:

1 - Plainly run a --backup-type=warm backup with no quiescing. This will imply not even 30 s. of downtime vs a nightmare that in the end will cause more downtime.

2 - Take a VSS snapshot prior to backing the VM up and delete it afterwards, also without quiescing in the virtual machine backup operation. This will ensure that you can discard the VSS snapshot and return to a stable configuration, should some I/O operation get trapped during the VM snapshot process.

Offline

Board footer