#1 2020-01-08 02:57:05

vscheuber
Member
Registered: 2019-11-18
Posts: 10

xsibackup removing snapshots?

Hi,

I'm feeling a bit unsure asking this but I have been very happy with XSIBackup Pro until I discovered this morning that all my manual snapshots of my VMs had disappeared. I have XSIBackup Pro setup to take full a backup of all VMs weekly and incremental backups of running VMs daily. I honestly can't remember whether I validated that snapshots were still there when I configured XSIBackup initially and whether I am just missing something that's logical to everybody else (that snapshots will be removed). So my question to you all here is: Is what I am seeing normal?

I also noticed that the number of VMs has doubled. It appears as if XSIBackup has duplicated all my VMs, the duplicates being named <original name>_XSIBAK and they seem to link to the files in my backup location.

VM Duplication

I don't remember seeing any errors reported in any of the daily job emails, so I am not sure what happened and where to look. Here are the two job definitions I am using:

001 - Full Backup

[root@esxi:/vmfs/volumes/f8282ebf-45634a14/xsi-dir/jobs] cat 001
"/vmfs/volumes/datastore11/xsi-dir/xsibackup" \
--backup-prog=Vmkfstools \
--snapshot=includememory \
--backup-point=/vmfs/volumes/datastore11/matterhorn/full \
--backup-type=All \
--backup-how=Hot \
--use-smtp=1 \
--mail-to=name@gmail.com \
--backup-id=001 \
--description="All VMs" \
--date-dir=yes \
--backup-room=5000 \
--exec=yes >> "/vmfs/volumes/datastore11/xsi-dir/var/logs/xsibackup.log"

002 - Incremental Backup

[root@esxi:/vmfs/volumes/f8282ebf-45634a14/xsi-dir/jobs] cat 002
"/vmfs/volumes/datastore11/xsi-dir/xsibackup" \
--backup-prog=OneDiff \
--snapshot=includememory \
--backup-point=/vmfs/volumes/datastore11/matterhorn/incremental \
--backup-type=Running \
--backup-how=Hot \
--use-smtp=1 \
--mail-to=name@gmail.com \
--backup-id=002 \
--description="Running VMs" \
--exec=yes >> "/vmfs/volumes/datastore11/xsi-dir/var/logs/xsibackup.log"

Thank you for your insights!

Cheers.
Volker

Offline

#2 2020-01-08 10:40:37

admin
Administrator
Registered: 2017-04-21
Posts: 1,366

Re: xsibackup removing snapshots?

Vmkfstools always consolidates snapshots into a unified base disk. If you want to preserve snapshots use some other backup program. This is plainly how Vmkfstools work, it creates a base disk from all the snapshots below the one you point Vmkfstools at. It's clearly stated in: https://33hops.com/xsibackup-using-backup-programs.html

To preserve snapshots you may use Onediff (this will create a replica, not a backup though), XSIDiff or Rsync (only over IP).

Offline

#3 2020-01-08 15:13:05

vscheuber
Member
Registered: 2019-11-18
Posts: 10

Re: xsibackup removing snapshots?

I think my original statement was unclear. I did not expect snapshots to survive in the backups. My report is about all my snapshots in the original VMs being lost. So, this is what's happened:

Before:

Original                    Backup
- VM1                       - VM1
  - Snapshot1
  - Snapshot2            
- VM2                       - VM2
  - Snapshot1

Now:

Original                    Backup
- VM1                       - VM1           
- VM2                       - VM2

I never expected snapshots to make it into my backups. But I did not expect to lose my original snapshots in my running VMs.

Any comments on the duplication of VMs? I did not import VMs from the backup location. Is that a side effect of XSIBackup? A malfunction?

Last edited by vscheuber (2020-01-08 15:27:55)

Offline

#4 2020-01-08 17:44:08

admin
Administrator
Registered: 2017-04-21
Posts: 1,366

Re: xsibackup removing snapshots?

(c)XSIBackup just creates a snapshot on top of any previously existing one and then deletes that very same snapshot, it doesn't take any action in regards to any previously existing one.

Remove --snapshot=includememory, this just adds the memory state of the backup snapshot which is useless 99,99% of the times.

Offline

#5 2020-01-08 18:20:54

vscheuber
Member
Registered: 2019-11-18
Posts: 10

Re: xsibackup removing snapshots?

Thanks, admin! Has anyone ever come across the loss of all their snapshots? I don't want to post my full backup logs here, but would you be willing to take a quick look and tell me if you can spot anything that would remove all the snapshots from my original VMs?

I do see sporadic error messages concerning snapshots, like this one:

---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:45|  [VM1] Starting backup (size is 34937M on 58362M file)
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:45|  XSIBackup will backup your VMs while they are running, so that users can continue to use the VM
2019-11-20T08:30:45|  while the backup is taking place. You can also run --backup-how=cold|warm
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:45|  Hot backup selected for VM: [VM1], will not be switched off
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:45|  [VM1] info: boot partition is MBR
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:46|  2      vMac                            [datastore10] vMac/vMac.vmx                                                     darwin16_64Guest        vmx-13                  
2019-11-20T08:30:47|  [VM1] info: quick size check...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:47|  [VM1] info: file [FRDP-flat.vmdk]...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:47|  [VM1] info: file size check | OK [ 59990000640 bytes | 59990000640 bytes ]
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:47|  [VM1] info: remote VMX file size 2858
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:47|  System disk CIDs 11c8d76f|11c8d76f
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:47|  [VM1] (c) OneDiff algorithm
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:47|  Mirror VM exists and the system disks CID are the same, starting OneDiff...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:47|  [VM1] info: length of .vmx file before OneDiff phase is 3018
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:48|  Snapshot & Quiescing
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:48|  [VM1] info: VMWare Tools detected, taking snapshot QUIESCED(false)...
2019-11-20T08:30:48|  [VM1] info: set argument --snapshot=doquiesce to quiesce your VMs
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:48|  [VM1] info: round 4
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:51|  20     Builder                         [datastore3] Builder/Builder.vmx                                               centos7_64Guest         vmx-14                  
2019-11-20T08:30:54|  [VM1] info: snapshot taken, quiescing status: 
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:54|  4      Windows 10 Desktop              [datastore10] Windows 10 Desktop/Windows 8.1 Desktop.vmx                        windows9_64Guest        vmx-11                  
2019-11-20T08:30:54|  INFO: found [VM1_XSIBAK] with Id: 21
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:55|  Initiating OneDiff differential backup...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:30:55|  [VM1] info: delta remains cleaned up at mirror VM [VM1_XSIBAK]
---------------------------------------------------------------------------------------------------------------------------------
[VM1] info: transfering snapshot file [FRDP_36-Snapshot20.vmsn]
[32m(c) XSIDiff 1.1.1.4 licensed: 
[0mA full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/datastore10/FRDP_36/FRDP_36-Snapshot20.vmsn
Writing to file /vmfs/volumes/datastore11/matterhorn/VM1/FRDP_36-Snapshot20.vmsn
Set input file offset to 0 bytes

100.00% done
Time taken: 1 seconds
Speed 1.00 mb/s
Overall speed 0.03 mb/s
---------------------------------------------------------------------------------------------------------------------------------
[VM1] info: transfering snapshot file [FRDP-000001.vmdk]
[32m(c) XSIDiff 1.1.1.4 licensed: 
[0mA full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/datastore10/FRDP_36/FRDP-000001.vmdk
Writing to file /vmfs/volumes/datastore11/matterhorn/VM1/FRDP-000001.vmdk
Set input file offset to 0 bytes

100.00% done
Time taken: 1 seconds
Speed 1.00 mb/s
Overall speed 0.00 mb/s
---------------------------------------------------------------------------------------------------------------------------------
[VM1] info: transfering snapshot file [FRDP-000001-delta.vmdk]
[32m(c) XSIDiff 1.1.1.4 licensed: 
[0mA full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/datastore10/FRDP_36/FRDP-000001-delta.vmdk
Writing to file /vmfs/volumes/datastore11/matterhorn/VM1/FRDP-000001-delta.vmdk
Set input file offset to 0 bytes

0.09% done
...
100.00% done
2019-11-20T08:31:14|  Needed room: 1501 Gb.
2019-11-20T08:31:14|  Sparse size: 1787 Gb.

Time taken: 18 seconds
Speed 64.06 mb/s
Overall speed 64.01 mb/s
2019-11-20T08:31:14|  Available room: 13934 Gb.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------
2019-11-20T08:31:15|  [VM1] info: length of remote .vmx file after copying it is 2857
2019-11-20T08:31:15|  [VM1] info: length of remote .vmx after replacing name is 2864
2019-11-20T08:31:15|  [VM1] info: name changed to [VM1_XSIBAK]
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:15|  Removing hot snapshot & merging data
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:16|  [VM1] Starting backup (size is 34937M on 58362M file)
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:16|  XSIBackup will backup your VMs while they are running, so that users can continue to use the VM
2019-11-20T08:31:16|  while the backup is taking place. You can also run --backup-how=cold|warm
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:16|  Hot backup selected for VM: [VM1], will not be switched off
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:16|  [VM1] info: boot partition is MBR
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:17|  Removing snapshots, please wait...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:19|  Syncronizing config files
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:22|  Memory will be included in the snapshot
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:22|  [VM1] info: VMWare Tools detected, taking snapshot QUIESCED(false)...
2019-11-20T08:31:22|  [VM1] info: set argument --snapshot=doquiesce to quiesce your VMs
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:25|  [VM1] error CLCRESN1: cannot create snapshot, details: Create Snapshot:
2019-11-20T08:31:25|  Create snapshot failed
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:26|  Backing up virtual disks...
---------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:40|  [VM2] Starting backup (size is 179202M on 179190M file)
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:40|  XSIBackup will backup your VMs while they are running, so that users can continue to use the VM
2019-11-20T08:31:40|  while the backup is taking place. You can also run --backup-how=cold|warm
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:40|  Hot backup selected for VM: [VM2], the VM is off though
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:41|  [VM2] info: boot partition is MBR
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:41|  [VM2] info: Win64 OS (windows8Server64) detected on MBR, applying Windows algorithm
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:42|  Removing snapshots, please wait...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:42|  Syncronizing config files
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:45|  Backing up virtual disks...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:31:45|  scsi0:0.fileName = "VM2-000004.vmdk"
---------------------------------------------------------------------------------------------------------------------------------
Destination disk format: VMFS thin-provisioned
Cloning disk '/vmfs/volumes/datastore10/VM2/VM2-000004.vmdk'...

Clone: 6% done.2019-11-20T08:34:19|  [VM1] info: OneDiff backup differential data (20) merged with base data
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:34:20|  [VM1] info: diff mirror VM reloaded
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:34:20|  [VM1] info: deleting snapshot at [VM1_XSIBAK] Id 21...
2019-11-20T08:34:21|  [VM1] info: all snapshots removed
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:34:21|  [VM1] info: Updating CID at [/vmfs/volumes/datastore11/matterhorn/VM1/FRDP.vmdk]
2019-11-20T08:34:21|  [VM1] info: finished OneDiff backup, CID updated
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:34:21|  [VM1] info: performing quick check...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:34:21|  [VM1] info: file [FRDP-flat.vmdk]...
2019-11-20T08:34:21|  [VM1] error: different sizes for [FRDP-flat.vmdk] [ 59990000640 <ne>  ]
2019-11-20T08:34:21|  [VM1] error DIFQCSIZ: file size check KO...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:34:21|  [VM1] info: comparing first 500M of [FRDP-flat.vmdk]...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:34:24|  [VM1] error DIFQMSH2: empty first 500M remote (inexistent file?)...
2019-11-20T08:34:24|  [VM1] info: (source) first 500M hash is: 39c15b3ad04137cda1756bd2b6cadc59a8f0379a
2019-11-20T08:34:24|  [VM1] info: (target) first 500M hash is: da39a3ee5e6b4b0d3255bfef95601890afd80709
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:34:24|  [VM1] error DIFQMSH4: first 500M mistmatch...
---------------------------------------------------------------------------------------------------------------------------------
2019-11-20T08:34:24|  [VM1] measure DIFRMVMX: took measure, renamed remote .vmx file to reinitialize OneDiff

Looking at the dates of the errors, this might have well happened while I was debugging this issue: DOW crontab entries execute daily and had the full backup and the incremental backup jobs run at the same time. There were no more errors after that. Could the two jobs run in parallel have cause the loss of snapshots on the original VMs?

Offline

#6 2020-01-13 17:54:45

admin
Administrator
Registered: 2017-04-21
Posts: 1,366

Re: xsibackup removing snapshots?

The output you are posting contains some data mismatch which has in turn caused a (c)Onediff image to be reset. A hash mismatch is usually an advanced warning on silent data corruption, maybe some warn out disk.

https://33hops.com/xsibackup-disk-check … ption.html

The code

2019-11-20T08:34:20|  [VM1] info: deleting snapshot at [VM1_XSIBAK] Id 21...
2019-11-20T08:34:21|  [VM1] info: all snapshots removed

Is obviously referring to [VM1_XSIBAK] Id 21 which is the mirror VM in the target system and has nothing to do with the snapshots on your primary data. That code indicates that the remote snapshot was deleted and the differential data has been integrated into the remote base disks.

In any case (c)Onediff prepares the VM to maintain a differential snapshot that holds the data in between backup cycles, so yes, you could have deleted some previous snapshots (that is coalesced the snapshot data into the base disks) in the original VM in preparation for the differential snapshot in case you would not know how (c)Onediff operates and you would have used it in a VM containing some previously stacked snapshots.

https://33hops.com/xsibackup-pro-onediff.html

Offline

Board footer