©XSIBackup-Free: Free Backup Software for ©VMWare ©ESXi

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software


You are not logged in.

#1 2021-02-08 08:46:53

Nils
Member
Registered: 2021-02-08
Posts: 8

safe local backup

Hi,
I use XSIBackup PRO and there are many options. ;-)

So, what ist the safest --backup-prog (and fast) way to backup a 800GB data.vmdk with a running (hot) Windows Server and many databases
from the local datastore1 to a local datastore2 on the same host?

--backup-prog vmkfstools onediff or xsitools?

It is safer to use the warm mode?

Thanks for help,
Nils

Offline

#2 2021-02-08 15:03:51

admin
Administrator
Registered: 2017-04-21
Posts: 1,821

Re: safe local backup

The fastest way is OneDiff, as it performs an instant differential backup. Due to its nature (not all data is copied everytime), it depends on the ESXi snapshot subsystem to ensure the backend data is consistent. You can verify that by adding --certify-backup to the backup job, so that the whole disks are checksumed and compared.

Now the additional challenge is that you are hosting a database server on the Windows server. You don't say much about your set up and complex scenarios can't be backed by simple laconic answers. We will asume that it is some version of MS SQL Server.

The next question to pose yourself is: "is the database server iddle during the backup windows?". If so, your problems is greatly simplified as you just have to stop the database service during the backup window and turn it back on again once the backup is complete, which is a really short time if you use Onediff.

If your DB server is busy during the backup windows and you have to keep it in production, then your problem complicates exponentially. Not only because you have to make sure you quiesce the DB server properly but because quiescing MS SQL Server can become quite complicated when compared to other database servers.

Before even attempting to backup your server, you should make sure you can take quiesced snapshots manually while the DB server is under heavy load. You can set up some script to create a table and add a few registers per second while you perform the quiescing tests.

While your DB server is under write load, take a quiesced snapshot from the ESXi snapshot manager and see if everything goes well and no errors are returned.

Depending on the MS SQL Server version that you are using you may need to install additional services for the Windows guest to be able to properly quiesce the database server. The basic picture is the following:

ESXi requests the guest OS to be quiesced.

If you have VMWare Tools installed (essential), the snapshot request will propagate to the guest OS and the database server will be asked to perform a controlled stop. In the end quiescing is similar to a service restart, in fact you will experience a short service cut while the DB is quiesced.

When you perform a service stop or restart on any database service, the process that controlls the service shutdown has to make sure that any pending I/O operations are performed consistently before actually bringing the service down. From the database perspective this means that any pending atomic operation or transaction is completed and only after this pending operations are performed and ensured to have been written consistently, the service can be shut down.

When you quiesce the database server the first part of the operation is the same, the only difference, which makes things more complicated is that traffic keeps on coming in, and so, quiescing consists not only in finishing pending I/O operations, but keeping new ones in a buffer, so that they are commited once the ESXi snapshot has been created and is ready to receive data.

At the end of the process, any ongoing ACID operations at the time the snapshot request comes in are flushed to the base disk, be it a -flat.vmdk file or a previous snapshot. and the subsequent ones are written to the newly created snapshot. The difference is that quiescing takes care to split things right, so that the database files are always consistent.

Per instance, suppose that you have some huge insert of some blob data going on when you request the guest to quiesce. VMWare Tools and the rest of implied services: Virtual Disk, VSS Services, VSS Services especific to SQL Server, etc... will take care to wait for the blob insert to finish and to hold the subsequent requests until the snapshot is up and ready to receive data.

Now the thing is: which services do you need to have installed and ready for this DB quiescing to work flawlessly?. As said, that will depend on the MS SQL Server version and how it's set up. You will find specific documentation in the database server manual.

Workarounds.

There are ways to just use general procedures to make sure your DB is consistent, below just a few:

- Perform a warm backup: this will stop the guest, take a snapshot from a shut down state and restart the VM. This ensures consistency at the price of just a few seconds of stop.

- Perform a DB backup before the VM backup and don't quiesce: this will ensure you have a consistent backup that you will have to restore after the VM has been restored.

- Use custom scripts to stop the database service just before the snapshot is taken and start it back again when the snapshot is ready:
Running custom quiescing scripts inside VM

Offline

#3 2021-02-09 14:52:52

Nils
Member
Registered: 2021-02-08
Posts: 8

Re: safe local backup

Thanks you for your help.

A new problem.
I backup a 8Gb Linux vm. ESXI 6.7u2

Config:
"/vmfs/volumes/datastore1/backup/XSIBackup-Pro/xsibackup" \
--backup-prog=OneDiff \
--backup-point=/vmfs/volumes/datastore2/vm/ \
--backup-type=Custom \
--backup-vms="Routerneu" \
--backup-how=Hot \
--backup-id=1 \
--description="Routerneu-1" \
--exec=yes >> "/vmfs/volumes/datastore1/backup/XSIBackup-Pro/var/logs/xsibackup.log"

The first backup is ok without any error.

In the vm filesystem there now new Snapshots with
Routerneu-000001-sesparse.vmdk
Routerneu-000001.vmdk

The second backup brings errors: CPERROR2 error: (c) XSIDiff 2.0.0.0 not licensed (20.00 GB limit)

What is wrong?



My log file:
2021-02-09T14:41:53|  [Routerneu] Starting backup (size is 6168M on 6188M file)
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:53|  XSIBackup will backup your VMs while they are running, so that users can continue to use the VM
2021-02-09T14:41:53|  while the backup is taking place. You can also run --backup-how=cold|warm
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:53|  Hot backup selected for VM: [Routerneu], will not be switched off
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:54|  [Routerneu] info: boot partition is MBR
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:56|  [Routerneu] info: quick size check...
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:56|  [Routerneu] info: file [Routerneu-flat.vmdk]...
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:56|  [Routerneu] info: file size check | OK [ 6442450944 bytes | 6442450944 bytes ]
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:56|  [Routerneu] info: remote VMX file size 3008
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:56|  System disk CIDs 8db41b57|8db41b57
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:56|  [Routerneu] (c) OneDiff algorithm
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:56|  Mirror VM exists and the system disks CID are the same, starting OneDiff...
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:56|  [Routerneu] info: length of .vmx file before OneDiff phase is 3007
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:57|  Snapshot & Quiescing
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:57|  [Routerneu] info: VMWare Tools detected, taking snapshot QUIESCED(false)...
2021-02-09T14:41:57|  [Routerneu] info: set argument --snapshot=doquiesce to quiesce your VMs
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:41:57|  [Routerneu] info: round 2
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:03|  [Routerneu] info: snapshot taken, quiescing status:
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:04|  INFO: found [Routerneu_XSIBAK] with Id: 8
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:04|  Initiating OneDiff differential backup...
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:04|  [Routerneu] info: delta remains cleaned up at mirror VM [Routerneu_XSIBAK]
---------------------------------------------------------------------------------------------------------------------------------
[Routerneu] info: transfering snapshot file [Routerneu-Snapshot15.vmsn]
A full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/datastore1/vm/Routerneu/Routerneu-Snapshot15.vmsn
Writing to file /vmfs/volumes/datastore2/vm/Routerneu/Routerneu-Snapshot15.vmsn
Set input file offset to 0 bytes
100.00% done
Time taken: 1 seconds
Speed 1.00 mb/s
Overall speed 0.03 mb/s
2021-02-09T14:42:05|  [Routerneu] CPERROR2 error: (c) XSIDiff 2.0.0.0 not licensed (20.00 GB limit)
2021-02-09T14:42:05|
---------------------------------------------------------------------------------------------------------------------------------
[Routerneu] info: transfering snapshot file [Routerneu-000001.vmdk]
A full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/datastore1/vm/Routerneu/Routerneu-000001.vmdk
Writing to file /vmfs/volumes/datastore2/vm/Routerneu/Routerneu-000001.vmdk
Set input file offset to 0 bytes
100.00% done
Time taken: 1 seconds
Speed 1.00 mb/s
Overall speed 0.00 mb/s
2021-02-09T14:42:06|  [Routerneu] CPERROR2 error: (c) XSIDiff 2.0.0.0 not licensed (20.00 GB limit)
2021-02-09T14:42:06|
---------------------------------------------------------------------------------------------------------------------------------
[Routerneu] info: transfering snapshot file [Routerneu-000001-sesparse.vmdk]
A full file copy will be performed, no meta-data available
Reading from file /vmfs/volumes/datastore1/vm/Routerneu/Routerneu-000001-sesparse.vmdk
Writing to file /vmfs/volumes/datastore2/vm/Routerneu/Routerneu-000001-sesparse.vmdk
Set input file offset to 0 bytes
100.00% done
Time taken: 2 seconds
Speed 22.50 mb/s
Overall speed 22.00 mb/s
2021-02-09T14:42:09|  [Routerneu] CPERROR2 error: (c) XSIDiff 2.0.0.0 not licensed (20.00 GB limit)
2021-02-09T14:42:09|
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:10|  [Routerneu] info: length of remote .vmx file after copying it is 3007
2021-02-09T14:42:10|  [Routerneu] info: length of remote .vmx after replacing name is 3014
2021-02-09T14:42:10|  [Routerneu] info: name changed to [Routerneu_XSIBAK]
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:10|  Removing hot snapshot & merging data
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:12|  [Routerneu] info: OneDiff backup differential data (15) merged with base data
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:12|  [Routerneu] info: diff mirror VM reloaded
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:12|  [Routerneu] info: deleting snapshot at [Routerneu_XSIBAK] Id 8...
2021-02-09T14:42:14|  [Routerneu] info: all snapshots removed
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:14|  [Routerneu] info: Updating CID at [/vmfs/volumes/datastore2/vm/Routerneu/Routerneu.vmdk]
2021-02-09T14:42:14|  [Routerneu] info: finished OneDiff backup, CID updated
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:15|  [Routerneu] info: performing quick check...
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:15|  [Routerneu] info: file [Routerneu-flat.vmdk]...
2021-02-09T14:42:15|  [Routerneu] info: file size check | OK [ 6442450944B | 6442450944B ]
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:15|  [Routerneu] info: comparing first 500M of [Routerneu-flat.vmdk]...
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:25|  [Routerneu] info: (source) first 500M hash is: 2e896d121f913123082153413478274083269e4c
2021-02-09T14:42:25|  [Routerneu] info: (target) first 500M hash is: 2e896d121f913123082153413478274083269e4c
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:25|  [Routerneu] info: first 500M check | OK [ 2e896d121f913123082153413478274083269e4c ]
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:25|  [Routerneu] info: [Routerneu-flat.vmdk] not a thin disk or FS not VMFS, used size check skipped
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:41|  The ESXi configuration was saved to "/vmfs/volumes/datastore2/vm"
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:41|  Errors detected in backup, check logs
---------------------------------------------------------------------------------------------------------------------------------
2021-02-09T14:42:41|  Backup finished
2021-02-09T14:42:41|  Tip: no chained backups scheduled, set --on-success and/or --on-error arguments to chain a backup
---------------------------------------------------------------------------------------------------------------------------------

Offline

#4 2021-02-09 16:25:36

admin
Administrator
Registered: 2017-04-21
Posts: 1,821

Re: safe local backup

We believe you own a Pro license. Then why don't you generate your license.key and install it to the root of your installation dir?.
Login to your user area and upload your request.key file (from the same installation root dir), then download the license.key once it is generated and follow the instructions in the key manager zone.

If you are trying to expose that that message should not be thrown because the VM is 8 GB then you are not using Pro but Free edition.

Apart from that, using Onediff to backup an 8GB VM is counter productive, as it will probably take less to perform a full backup by using vmkfstools or XSITools than to risk exchanging a differential snapshot, especially since you don't even seem to be hosting the VM on a VFMS volume

Your log does not seem to be raising any errors apart from that CPERROR2. You may use ondediff:f to force using Rsync instead of XSIDiff.

Offline

Board footer