#1 2020-06-14 14:46:54

sphen
Member
Registered: 2020-06-13
Posts: 45

replica issues with thin disks - seemingly thick

Good morning.

In my first test of DC to replicate a VM over IP, it seems to be creating a thick version of the vm at the remote site. Looking at the docs it seems that this should not be the case and wonder where I made a mistake.

the replica command being used:

./xsibackup --replica "VMs(Radius)" root@X.X.X.X:22:/volume1/backup --ssh-ciphers=aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc

so far my progress is:

Backup start date: 2020-06-14 13:42:37
-----------------------------------------------------------------------------------------------------------
2020-06-14 13:42:37 | Backing up 15 files, total size is 64425634867
-----------------------------------------------------------------------------------------------------------
    NUMBER	                                                   FILE             SIZE       PROGRESS
-----------------------------------------------------------------------------------------------------------
     1/15                                              Radius-flat.vmdk         60.00 GB | Done   0.00%
-----------------------------------------------------------------------------------------------------------
::: detail :::  55.37% done | block    34022 out of   61440                              | Done  55.37%

looking at the source flat file specifically with du:

15.4G	Radius-flat.vmdk

looking st the current destination flat file:

34G	Radius-flat.vmdk.tmp

What am i missing here? Thank you.

Offline

#2 2020-06-15 12:42:42

admin
Administrator
Registered: 2017-04-21
Posts: 1,364

Re: replica issues with thin disks - seemingly thick

We believe that has more to do with your remote OS, FS and du implementation than with DC binary. What it does is to create an empty sparse file, then it fills it with the non-zero data.

Even if you don't host your VM in a VMFS volume, (c)XSIBackup-DC will still jump over zeros in the replica, thus the resulting replicated -flat.vmdk file is always a sparse (thin in (c)VMWare's terminology) file.

Try to run this alternative command and see if that returns the non-zero data on the leftmost column.

ls -lash Radius-flat.vmdk

Offline

#3 2020-06-16 02:38:52

sphen
Member
Registered: 2020-06-13
Posts: 45

Re: replica issues with thin disks - seemingly thick

Sadly ls gave the same result. Also judging on data speed and time taken I believe it transferred the whole amount:

60G -rw-r--r-- 1 admin root 60G Jun 15 22:21 Radius-flat.vmdk

More troubling and maybe something indicating a different issue my setup has that is causing this, the replica vmdk seems to be unusable. I brought that replica online in the remote esxi environment and the disk was indicated to be unformatted. I am unsure if this would really work, but i put the vmdk-flat file through less it seems blank.

When I do the same command but --backup instead of --replica, the backup goes very fast, and creates an extremely small repository. I was able to get the same result over ssh or local NFS mount. Doing a restore of the backup creates a blank 60 GB vmdk as well.

I have included the output of various commands below to help in figuring out what is going wrong? These commands were directed at a much smaller VM for quicker testing - thin provisioned and had the same result. The host is running ESXi 6.7.0 (Build 8169922).

/vmfs/volumes/SSD/XSIBackup-DC/xsibackup --replica "VMs(Auvik)" root@X.X.X:22:/volume1/vertical --ssh-ciphers=aes128-ctr

replica:

|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-----------------------------------------------------------------------------------------------------------
License: unlicensed trial version
-----------------------------------------------------------------------------------------------------------
SSH: Mon Jun 15 21:33:06 EDT 2020
-----------------------------------------------------------------------------------------------------------
Remote system: linux
-----------------------------------------------------------------------------------------------------------
User defined ciphers: aes128-ctr
-----------------------------------------------------------------------------------------------------------
PID: 2100191, Running job as: root
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC replicating data to /volume1/vertical
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: Auvik
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : Auvik (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
New backup/replica: Auvik | folder
-----------------------------------------------------------------------------------------------------------
Backup start date: 2020-06-16 01:44:28
-----------------------------------------------------------------------------------------------------------
2020-06-16 01:44:28 | Backing up 24 files, total size is 10872008991
-----------------------------------------------------------------------------------------------------------
    NUMBER	                                                   FILE             SIZE       PROGRESS
-----------------------------------------------------------------------------------------------------------
     1/24                                               Auvik-flat.vmdk          8.00 GB | Done   0.00%
-----------------------------------------------------------------------------------------------------------
     2/24                                                    Auvik.vmdk        565.00 B  | Done  79.01%
-----------------------------------------------------------------------------------------------------------
     3/24                                             Auvik_1-flat.vmdk       1024.00 KB | Done  79.01%
-----------------------------------------------------------------------------------------------------------
     4/24                                                  Auvik_1.vmdk        560.00 B  | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     5/24                                                   Auvik.nvram          8.48 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     6/24                                                  vmware-8.log        201.22 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     7/24                                                  vmware-9.log        196.03 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     8/24                                                Auvik.vmsd.tmp         43.00 B  | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     9/24                                                 Auvik.vmx.tmp          2.53 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
    10/24                                                 vmware-10.log        193.21 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
    11/24                                                 vmware-11.log        116.99 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
    12/24                                                 vmware-12.log        223.09 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    13/24	                            vmx-Auvik-3446253671-1.vswp   [open excluded]| Done  79.03%
-----------------------------------------------------------------------------------------------------------
    14/24	                                          Auvik.vmx.lck   [open excluded]| Done  79.03%
-----------------------------------------------------------------------------------------------------------
    15/24                                                 vmware-13.log        168.58 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    16/24                                                    vmware.log        230.47 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    17/24	                                    Auvik-cd69a867.vswp   [open excluded]| Done  79.03%
-----------------------------------------------------------------------------------------------------------
    18/24                                          Auvik-Snapshot4.vmsn         19.16 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    19/24	                                Auvik-000001-delta.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    20/24	                                      Auvik-000001.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    21/24	                              Auvik_1-000001-delta.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    22/24	                                    Auvik_1-000001.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    23/24                                                     Auvik.vmx          2.54 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    24/24                                                    Auvik.vmsd        501.00 B  | Done  79.03%
-----------------------------------------------------------------------------------------------------------
Total size:              8194 MB                                                        | Done 100.00%
-----------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-----------------------------------------------------------------------------------------------------------
Backup end date: 2020-06-16 02:03:50
-----------------------------------------------------------------------------------------------------------
Time taken:     1162 sec.
-----------------------------------------------------------------------------------------------------------
Total time:     1162 sec.
-----------------------------------------------------------------------------------------------------------
Full file speed:	                                                                     7.05 mb/s
-----------------------------------------------------------------------------------------------------------
Real data speed:	                                                                     7.92 mb/s
-----------------------------------------------------------------------------------------------------------
Item backup completed
-----------------------------------------------------------------------------------------------------------
Data processing completed successfully
-----------------------------------------------------------------------------------------------------------
Removed <tmp> dir        OK
-----------------------------------------------------------------------------------------------------------
Unlocked backup          OK
-----------------------------------------------------------------------------------------------------------
SSH session was closed   OK
-----------------------------------------------------------------------------------------------------------
Removed PID              OK
-----------------------------------------------------------------------------------------------------------

source:

[root@esxi2:/vmfs/volumes/662879e1-2d0c7453-0000-000000000000/Auvik] du -ksh Auvik-flat.vmdk 
4.1G	Auvik-flat.vmdk
[root@esxi2:/vmfs/volumes/662879e1-2d0c7453-0000-000000000000/Auvik] ls -lash Auvik-flat.vmdk 
4290308 -rw-------    1 root     root        8.0G Jun 16 02:33 Auvik-flat.vmdk

dest:

sh-4.3# du -ksh Auvik-flat.vmdk 
8.0G	Auvik-flat.vmdk
sh-4.3# ls -lash Auvik-flat.vmdk 
8.0G -rw-r--r-- 1 root root 8.0G Jun 15 21:56 Auvik-flat.vmdk

backup:

[root@esxi2:~] /vmfs/volumes/SSD/XSIBackup-DC/xsibackup --backup "VMs(Auvik)" root@X.X.X:22:/volume1/vertical/xsi --ssh-ciphers=aes128-ctr
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-----------------------------------------------------------------------------------------------------------
License: unlicensed trial version
-----------------------------------------------------------------------------------------------------------
SSH: Mon Jun 15 21:53:01 EDT 2020
-----------------------------------------------------------------------------------------------------------
Remote system: linux
-----------------------------------------------------------------------------------------------------------
User defined ciphers: aes128-ctr
-----------------------------------------------------------------------------------------------------------
PID: 2100963, Running job as: root
-----------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-----------------------------------------------------------------------------------------------------------
Block size is 1048576 bytes (1024.00 KB)
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC setting repository at /volume1/vertical/xsi
-----------------------------------------------------------------------------------------------------------
.xsitools file created at: root@X.X.X.X:22:/volume1/vertical/xsi/.xsitools
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: Auvik
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : Auvik (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
New backup/replica: Auvik | folder
-----------------------------------------------------------------------------------------------------------
Backup start date: 2020-06-16 02:04:27
-----------------------------------------------------------------------------------------------------------
2020-06-16 02:04:27 | Backing up 24 files, total size is 10872042917
-----------------------------------------------------------------------------------------------------------
    NUMBER	                                                   FILE             SIZE       PROGRESS
-----------------------------------------------------------------------------------------------------------
     1/24                                               Auvik-flat.vmdk          8.00 GB | Done   0.00%
-----------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block    8192 out of    8192                               | Done  79.01%
-----------------------------------------------------------------------------------------------------------
     2/24                                                    Auvik.vmdk        565.00 B  | Done  79.01%
-----------------------------------------------------------------------------------------------------------
     3/24                                             Auvik_1-flat.vmdk       1024.00 KB | Done  79.01%
-----------------------------------------------------------------------------------------------------------
     4/24                                                  Auvik_1.vmdk        560.00 B  | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     5/24                                                   Auvik.nvram          8.48 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     6/24                                                  vmware-8.log        201.22 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     7/24                                                  vmware-9.log        196.03 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     8/24                                                Auvik.vmsd.tmp         43.00 B  | Done  79.02%
-----------------------------------------------------------------------------------------------------------
     9/24                                                 Auvik.vmx.tmp          2.53 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
    10/24                                                 vmware-10.log        193.21 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
    11/24                                                 vmware-11.log        116.99 KB | Done  79.02%
-----------------------------------------------------------------------------------------------------------
    12/24                                                 vmware-12.log        223.09 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    13/24	                            vmx-Auvik-3446253671-1.vswp   [open excluded]| Done  79.03%
-----------------------------------------------------------------------------------------------------------
    14/24	                                          Auvik.vmx.lck   [open excluded]| Done  79.03%
-----------------------------------------------------------------------------------------------------------
    15/24                                                 vmware-13.log        168.58 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    16/24                                                    vmware.log        263.37 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    17/24	                                    Auvik-cd69a867.vswp   [open excluded]| Done  79.03%
-----------------------------------------------------------------------------------------------------------
    18/24                                          Auvik-Snapshot5.vmsn         19.16 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    19/24	                                Auvik-000001-delta.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    20/24	                                      Auvik-000001.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    21/24	                              Auvik_1-000001-delta.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    22/24	                                    Auvik_1-000001.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    23/24                                                     Auvik.vmx          2.54 KB | Done  79.03%
-----------------------------------------------------------------------------------------------------------
    24/24                                                    Auvik.vmsd        500.00 B  | Done  79.03%
-----------------------------------------------------------------------------------------------------------
Total size:              8194 MB                                                        | Done 100.00%
-----------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-----------------------------------------------------------------------------------------------------------
Backup end date: 2020-06-16 02:04:46
-----------------------------------------------------------------------------------------------------------
Time taken:       19 sec.
-----------------------------------------------------------------------------------------------------------
Total time:       19 sec.
-----------------------------------------------------------------------------------------------------------
Full file speed:	                                                                   431.28 mb/s
-----------------------------------------------------------------------------------------------------------
Real data speed:	                                                                     2.36 mb/s
-----------------------------------------------------------------------------------------------------------
Item backup completed
-----------------------------------------------------------------------------------------------------------
Differential blocks were added to the .blocklog database
-----------------------------------------------------------------------------------------------------------
Data processing completed successfully
-----------------------------------------------------------------------------------------------------------
Removed <tmp> dir        OK
-----------------------------------------------------------------------------------------------------------
Unlocked backup          OK
-----------------------------------------------------------------------------------------------------------
SSH session was closed   OK
-----------------------------------------------------------------------------------------------------------
Removed PID              OK
-----------------------------------------------------------------------------------------------------------

destination repo info:

sh-4.3# xsibackup --info /volume1/vertical/xsi
Missing SSL key pair, generating... 
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: Linux, Kernel 4 Major 4 Minor 59 Patch 0
-----------------------------------------------------------------------------------------------------------
License: unlicensed trial version
-----------------------------------------------------------------------------------------------------------
PID: 30509, Running job as: root
-----------------------------------------------------------------------------------------------------------
Deduplicated repository information
-----------------------------------------------------------------------------------------------------------
Compression: 1
Block size: 1048576
Block count: 15
-----------------------------------------------------------------------------------------------------------
Retrieving block information...
-----------------------------------------------------------------------------------------------------------
                    |          Real data   |            Hosted data   |
-----------------------------------------------------------------------------------------------------------
Block number        |                 15   |                   8208   |
Data volume         |          362.69 KB   |                8.00 GB   |
Compression         |             100.00%   |                      -   |
-----------------------------------------------------------------------------------------------------------
Removed <tmp> dir        OK
-----------------------------------------------------------------------------------------------------------
Removed PID              OK
-----------------------------------------------------------------------------------------------------------

Thanks for the help - I really would love to get this to work.

Offline

#4 2020-06-16 17:48:29

admin
Administrator
Registered: 2017-04-21
Posts: 1,364

Re: replica issues with thin disks - seemingly thick

Sparseness depends on the OS kernel and FS, nothing to do with (c)XSIBackup-DC.
15 blocks is same as nothing.

All you are describing is really awkward. Please contact support for some personalized help, we would need to reproduce your issue by using the same target OS. This doesn't look like something that could be generalized or help others until we know the details of the target OS.

Offline

#5 2020-06-16 17:55:19

sphen
Member
Registered: 2020-06-13
Posts: 45

Re: replica issues with thin disks - seemingly thick

Thank you. Since I have not purchased yet at this point what would be the best point of contact for support? I see a contact form on the website is that what I should use? Thank you.

Offline

Board footer