Expected behavior of delta backup and speed

sphen · 2020-06-19 15:32:09

Hi there.

I am a new DC user and still trying to get my head around a few things.

I have successfully started using it to backup VMs to both ssh and NFS destinations. My question is in regards to the behavior of repeated --backup calls in regards to deduplication/block checks and speed.

As expected the first backup takes some time with the increase in speed mainly coming from deduplication/sparseness. Subsequent --backup calls on the same vm however seem to take longer than I would expect and I want to know if this behavior is normal within reason or if I have done something wrong.

Each time I call --backup to my destination repository it states:

New backup/replica:

Does this imply that it is treating this as a brand new backup which will not be utilizing previously uploaded blocks via .map files etc?

The comparison of blocks seems slow and I see data flowing from the server which implies that comparison/block check is not happening locally.

Increasing verbosity I do see each line showing something like:

Block 66, exists: 1, sha1str: 637376502371b9d41846eb086b964be061548207
::: detail :::   0.11% done | block      67 out of   61440                               | Done   0.09%

exists: 1

means its a match to a previously stored block I would presume?

From a second backup run from a vm that was previously backedup, it took nearly 2 hours to compare and complete while the server is not changing that much. The server has ~ 1 TB allocated to it and about ~ 470 GB of data utilization according to vCenter. I dont think this is expected behavior?

I just ran a quick test on a vm that was never backed up before. please see below the first and then second run. The second run was almost as long as the first. Any pointers on what to do/look for would be very helpful. Thank you!

FIRST RUN:

./xsibackup --backup "VMs(vSphere_Replication)" /vmfs/volumes/Offsite
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-----------------------------------------------------------------------------------------------------------
License: XXX
-----------------------------------------------------------------------------------------------------------
PID: 6912971, Running job as: root
-----------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-----------------------------------------------------------------------------------------------------------
Block size is 10485760 bytes (10.00 MB)
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC setting repository at /vmfs/volumes/Offsite
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: vSphere_Replication
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : vSphere_Replication (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
New backup/replica: vSphere_Replication | folder
-----------------------------------------------------------------------------------------------------------
Backup start date: 2020-06-19 14:45:59
-----------------------------------------------------------------------------------------------------------
2020-06-19 14:45:59 | Backing up 27 files, total size is 36746453608
-----------------------------------------------------------------------------------------------------------
    NUMBER	                                                   FILE             SIZE       PROGRESS
-----------------------------------------------------------------------------------------------------------
     1/27                             vSphere_Replication-62816573.hlog         92.00 B  | Done   0.00%
-----------------------------------------------------------------------------------------------------------
     2/27                                       vSphere_Replication.vmx          3.11 KB | Done   0.00%
-----------------------------------------------------------------------------------------------------------
     3/27                                 vSphere_Replication-flat.vmdk          9.00 GB | Done   0.00%
-----------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block     922 out of     922                               | Done  26.30%
-----------------------------------------------------------------------------------------------------------
     4/27                                      vSphere_Replication.vmdk        538.00 B  | Done  26.30%
-----------------------------------------------------------------------------------------------------------
     5/27                               vSphere_Replication_1-flat.vmdk         17.00 GB | Done  26.30%
-----------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block    1741 out of    1741                               | Done  75.98%
-----------------------------------------------------------------------------------------------------------
     6/27                                    vSphere_Replication_1.vmdk        541.00 B  | Done  75.98%
-----------------------------------------------------------------------------------------------------------
     7/27                                      vSphere_Replication.vmsd        553.00 B  | Done  75.98%
-----------------------------------------------------------------------------------------------------------
     8/27	              vmx-vSphere_Replication-2238554927-1.vswp   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
     9/27	                            vSphere_Replication.vmx.lck   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
    10/27	 hbr-persistent-state-RDID-5c50bca2-cd60-4837-8f60-c6f2   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
    11/27                                                  vmware-1.log        244.99 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    12/27                                      vSphere_Replication.vmx~          3.09 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    13/27                                     vSphere_Replication.nvram          8.48 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    14/27                                                  vmware-2.log        193.74 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    15/27                                                  vmware-3.log        201.37 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    16/27	 hbr-persistent-state-RDID-d7a5248e-f2ca-4a35-ad2f-c86d   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
    17/27                                                  vmware-4.log        234.00 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    18/27                                                  vmware-5.log        223.60 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    19/27                                                    vmware.log        213.05 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    20/27	                      vSphere_Replication-856da32f.vswp   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
    21/27                                  vSphere_Replication.vmsd.tmp          0.00 B  | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    22/27                                   vSphere_Replication.vmx.tmp          3.09 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    23/27                            vSphere_Replication-Snapshot1.vmsn         19.72 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    24/27	               vSphere_Replication-000001-sesparse.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    25/27	                        vSphere_Replication-000001.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    26/27	             vSphere_Replication_1-000001-sesparse.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    27/27	                      vSphere_Replication_1-000001.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
Total size:             26625 MB                                                        | Done 100.00%
-----------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-----------------------------------------------------------------------------------------------------------
Backup end date: 2020-06-19 15:08:10
-----------------------------------------------------------------------------------------------------------
Time taken:     1331 sec.
-----------------------------------------------------------------------------------------------------------
Total time:     1331 sec.
-----------------------------------------------------------------------------------------------------------
Full file speed:	                                                                    20.00 mb/s
-----------------------------------------------------------------------------------------------------------
Real data speed:	                                                                    60.43 mb/s
-----------------------------------------------------------------------------------------------------------
Item backup completed
-----------------------------------------------------------------------------------------------------------
Differential blocks were added to the .blocklog database
-----------------------------------------------------------------------------------------------------------
Data processing completed successfully
-----------------------------------------------------------------------------------------------------------
Removed <tmp> dir        OK
-----------------------------------------------------------------------------------------------------------
Unlocked backup          OK
-----------------------------------------------------------------------------------------------------------
Removed PID              OK
-----------------------------------------------------------------------------------------------------------

SECOND RUN:

./xsibackup --backup "VMs(vSphere_Replication)" /vmfs/volumes/Offsite
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software                    |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-----------------------------------------------------------------------------------------------------------
License: XXX
-----------------------------------------------------------------------------------------------------------
PID: 6913851, Running job as: root
-----------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-----------------------------------------------------------------------------------------------------------
Block size is 10485760 bytes (10.00 MB)
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC setting repository at /vmfs/volumes/Offsite
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: vSphere_Replication
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : vSphere_Replication (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
New backup/replica: vSphere_Replication | folder
-----------------------------------------------------------------------------------------------------------
Backup start date: 2020-06-19 15:09:17
-----------------------------------------------------------------------------------------------------------
2020-06-19 15:09:17 | Backing up 27 files, total size is 36746546791
-----------------------------------------------------------------------------------------------------------
    NUMBER	                                                   FILE             SIZE       PROGRESS
-----------------------------------------------------------------------------------------------------------
     1/27                             vSphere_Replication-62816573.hlog         92.00 B  | Done   0.00%
-----------------------------------------------------------------------------------------------------------
     2/27                                       vSphere_Replication.vmx          3.11 KB | Done   0.00%
-----------------------------------------------------------------------------------------------------------
     3/27                                 vSphere_Replication-flat.vmdk          9.00 GB | Done   0.00%
-----------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block     922 out of     922                               | Done  26.30%
-----------------------------------------------------------------------------------------------------------
     4/27                                      vSphere_Replication.vmdk        538.00 B  | Done  26.30%
-----------------------------------------------------------------------------------------------------------
     5/27                               vSphere_Replication_1-flat.vmdk         17.00 GB | Done  26.30%
-----------------------------------------------------------------------------------------------------------
::: detail ::: 100.00% done | block    1741 out of    1741                               | Done  75.98%
-----------------------------------------------------------------------------------------------------------
     6/27                                    vSphere_Replication_1.vmdk        541.00 B  | Done  75.98%
-----------------------------------------------------------------------------------------------------------
     7/27                                      vSphere_Replication.vmsd        553.00 B  | Done  75.98%
-----------------------------------------------------------------------------------------------------------
     8/27	              vmx-vSphere_Replication-2238554927-1.vswp   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
     9/27	                            vSphere_Replication.vmx.lck   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
    10/27	 hbr-persistent-state-RDID-5c50bca2-cd60-4837-8f60-c6f2   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
    11/27                                                  vmware-1.log        244.99 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    12/27                                      vSphere_Replication.vmx~          3.09 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    13/27                                     vSphere_Replication.nvram          8.48 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    14/27                                                  vmware-2.log        193.74 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    15/27                                                  vmware-3.log        201.37 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    16/27	 hbr-persistent-state-RDID-d7a5248e-f2ca-4a35-ad2f-c86d   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
    17/27                                                  vmware-4.log        234.00 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    18/27                                                  vmware-5.log        223.60 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    19/27                                                    vmware.log        257.01 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    20/27	                      vSphere_Replication-856da32f.vswp   [open excluded]| Done  75.98%
-----------------------------------------------------------------------------------------------------------
    21/27                                  vSphere_Replication.vmsd.tmp         43.00 B  | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    22/27                                   vSphere_Replication.vmx.tmp          3.09 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    23/27                            vSphere_Replication-Snapshot2.vmsn         19.72 KB | Done  75.98%
-----------------------------------------------------------------------------------------------------------
    24/27	               vSphere_Replication-000001-sesparse.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    25/27	                        vSphere_Replication-000001.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    26/27	             vSphere_Replication_1-000001-sesparse.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
    27/27	                      vSphere_Replication_1-000001.vmdk    [skipped excluded]
-----------------------------------------------------------------------------------------------------------
Total size:             26625 MB                                                        | Done 100.00%
-----------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-----------------------------------------------------------------------------------------------------------
Backup end date: 2020-06-19 15:26:30
-----------------------------------------------------------------------------------------------------------
Time taken:     1033 sec.
-----------------------------------------------------------------------------------------------------------
Total time:     1033 sec.
-----------------------------------------------------------------------------------------------------------
Full file speed:	                                                                    25.77 mb/s
-----------------------------------------------------------------------------------------------------------
Real data speed:	                                                                    73.35 mb/s
-----------------------------------------------------------------------------------------------------------
Item backup completed
-----------------------------------------------------------------------------------------------------------
Differential blocks were added to the .blocklog database
-----------------------------------------------------------------------------------------------------------
Data processing completed successfully
-----------------------------------------------------------------------------------------------------------
Removed <tmp> dir        OK
-----------------------------------------------------------------------------------------------------------
Unlocked backup          OK
-----------------------------------------------------------------------------------------------------------
Removed PID              OK
-----------------------------------------------------------------------------------------------------------

admin · 2020-06-20 12:28:14

©XSIBackup-DC has two major working modes for backups and replicas: local and remote, that is: passing a local path as the third argument or passing a remote path root@a.b.c.d:22:/some/remote/path

When you use the first mode (local), the software assumes your remote site is reachable within a reasonable local latency lapse, thus it queries each individual block before actually copying it or just adding its hash in case it already exists.

When you use the remote path syntax, it changes to a different algorithm by means of which the block maps are exchanged at the beginning of the process, then it starts traversing the disks in block size chunks to check individual hash and check whether it has to be sent or not.

If you mount some remote NFS server and the latency is high (non local), then you should not be using the local mode, but the remote, so that you get the block manifest on advance and perform all checksum comparison operation locally.

When you perform a local backup to a, let's say 50 ms latency remote host, you are adding 50ms to each individual block operation. Every mode has its reason of being, you just have to choose the most suitable one in each situation.

sphen · 2020-07-04 12:27:54

Now that I have been running things for awhile via cron with reporting I wanted to clarify my expectations towards what I am seeing. All VMs are now backing up via remote/ssh which from your great description above is the most efficient as it skips blocks in the most efficient way. Our VMs backup with a range of speed (all from the same datastore) ranging from 80 MB/sec -> 1000 MB/sec. Our largest VM which currently has ~ 4.6TB of used space is consistently taking 8+ hours to compare even when not much data is changing. I am really curious as to what your expectation on speed in an average scenario would be for this VM and if there is anything else we may do to improve it.

In this case I am going to a remote linux box via ssh so there is not vmkfstools, but i would imagine if i were going to a remote esxi host and running Free or Pro with onediff that it would be complete much faster. I understand that I am comparing quite different backup/replica scenarios though in this example.

Without getting lost in the weeds, do you think that 8+ hours is what i should expect for a VM of this size? What would you do in this scenario if you required a time efficient backup of such vm?

Thank you for your work.

admin · 2020-07-05 14:28:54

That's 575.00GB of real data (non zeros) per hour. Not bad, an average of 164.00 MB/s
It could be considerably faster if you used faster disks/controller

(c)XSIBackup-DC can jump over zeros, thus, if you backup a VM which is 4.00TB in size but only contains, let's say 20.00GB, it will be backed up really fast, as zeros will be ignored and from the second pass the 20GB will be traversed at read speed.

If your VM contains 4,6 TB of real data, nothing is going to keep you from having to walk through every byte. Most of the data will most probably not have changed, thus your effective speed on a (c)XSIBackup-DC backup will be close to the read speed you can achieve, as SHA-1 calculations are RAM fast even with a modest Intel CPU. That should yield over 200.00MB/S on enterprise grade HDs and about twice that on enterprise grade SSDs or NVMe, M2, ...

As you say (c)OneDiff can be much faster in this case, but you must not forget some facts: (c)OnedIff depends on the differential snapshot being integrated in the remote VM, that requires some ESXi host to replicate data. Apart from that, the integration of the snapshot may fail depending on the ESXi host conditions, while (c)XSIBackup-DC differential feature is completely independent of the target system. As (c)XSIBackup-DC walks through every byte, you can be sure that the resulting VM will be consistent, no matter what happens with the backup snapshot.

Apart from that, (c)XSIBackup-DC will only be sending data when it detects some block that has changed, thus, although it may take 8 h. to complete, all that it's doing most of the time is to read data.

sphen · 2020-07-07 13:20:13

Thank you - makes sense. I will check more into this storage array as I still believe that I should be getting faster speed from it. Your explanation makes sense however and I would have to account for this if i was trying to backup a larger VM. The zero block skipping has been working great as this disk is much larger than 4.6 TB.

Thanks.

admin · 2020-07-07 18:19:52

You are then playing in a mayor league. Fast NVMe, more RAM and 10GB NICs are a must in your case.

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

#1 2020-06-19 15:32:09

Expected behavior of delta backup and speed

#2 2020-06-20 12:28:14

Re: Expected behavior of delta backup and speed

#3 2020-07-04 12:27:54

Re: Expected behavior of delta backup and speed

#4 2020-07-05 14:28:54

Re: Expected behavior of delta backup and speed

#5 2020-07-07 13:20:13

Re: Expected behavior of delta backup and speed

#6 2020-07-07 18:19:52

Re: Expected behavior of delta backup and speed

Board footer