Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

You see that message when some remote replica performed via --backup-prog=Onediff is switched on. When you do so, you modify the seed that the Onediff differential algorithm created on the previous run and a new full replica must be completed to ensure that both ends are identical.

To avoid that from happening and allow Onediff to just add the instant differential data, just don't switch on the remote end replica in between Onediff cycles.

The fastest way is OneDiff, as it performs an instant differential backup. Due to its nature (not all data is copied everytime), it depends on the ESXi snapshot subsystem to ensure the backend data is consistent. You can verify that by adding [b]--certify-backup[/b] to the backup job, so that the whole disks are checksumed and compared.

Now the additional challenge is that you are hosting a database server on the Windows server. You don't say much about your set up and complex scenarios can't be backed by simple laconic answers. We will asume that it is some version of [b]MS SQL Server[/b].

The next question to pose yourself is: "is the database server iddle during the backup windows?". If so, your problems is greatly simplified as you just have to stop the database service during the backup window and turn it back on again once the backup is complete, which is a really short time if you use [b]Onediff[/b].

If your DB server is busy during the backup windows and you have to keep it in production, then your problem complicates exponentially. Not only because you have to make sure you quiesce the DB server properly but because quiescing [b]MS SQL Server[/b] can become quite complicated when compared to other database servers.

Before even attempting to backup your server, you should make sure you can take quiesced snapshots manually while the DB server is under heavy load. You can set up some script to create a table and add a few registers per second while you perform the quiescing tests.

While your DB server is under write load, take a quiesced snapshot from the ESXi snapshot manager and see if everything goes well and no errors are returned.

Depending on the [b]MS SQL Server[/b] version that you are using you may need to install additional services for the Windows guest to be able to properly quiesce the database server. The basic picture is the following:

ESXi requests the guest OS to be quiesced.

If you have VMWare Tools installed (essential), the snapshot request will propagate to the guest OS and the database server will be asked to perform a controlled stop. In the end quiescing is similar to a service restart, in fact you will experience a short service cut while the DB is quiesced.

When you perform a service stop or restart on any database service, the process that controlls the service shutdown has to make sure that any pending I/O operations are performed consistently before actually bringing the service down. From the database perspective this means that any pending atomic operation or transaction is completed and only after this pending operations are performed and ensured to have been written consistently, the service can be shut down.

When you quiesce the database server the first part of the operation is the same, the only difference, which makes things more complicated is that traffic keeps on coming in, and so, quiescing consists not only in finishing pending I/O operations, but keeping new ones in a buffer, so that they are commited once the ESXi snapshot has been created and is ready to receive data.

At the end of the process, any ongoing [b][url=https://en.wikipedia.org/wiki/ACID]ACID[/url][/b] operations at the time the snapshot request comes in are flushed to the base disk, be it a [b]-flat.vmdk[/b] file or a previous snapshot. and the subsequent ones are written to the newly created snapshot. The difference is that quiescing takes care to split things right, so that the database files are always consistent.

Per instance, suppose that you have some huge insert of some blob data going on when you request the guest to quiesce. VMWare Tools and the rest of implied services: Virtual Disk, VSS Services, VSS Services especific to SQL Server, etc... will take care to wait for the blob insert to finish and to hold the subsequent requests until the snapshot is up and ready to receive data.

Now the thing is: which services do you need to have installed and ready for this DB quiescing to work flawlessly?. As said, that will depend on the MS SQL Server version and how it's set up. You will find specific documentation in the database server manual.

[b]Workarounds[/b].

There are ways to just use general procedures to make sure your DB is consistent, below just a few:

- Perform a warm backup: this will stop the guest, take a snapshot from a shut down state and restart the VM. This ensures consistency at the price of just a few seconds of stop.

- Perform a DB backup before the VM backup and don't quiesce: this will ensure you have a consistent backup that you will have to restore after the VM has been restored.

- Use custom scripts to stop the database service just before the snapshot is taken and start it back again when the snapshot is ready:
[url=https://kb.vmware.com/s/article/1006671]Running custom quiescing scripts inside VM[/url]

Yes, you can run a [b]--check[/b] argument against a full repo or any of its subfolders, be it a full timestamped backup containing VMs or a single VM

Checks will be much faster with SSDs, as seek times are many orders of magnitude faster in SSDs than in HDs

Also: please note that although XSIBackup will not keep you from performing a [b]--check[/b] action from the ESXi host (in case of NFS datastores), you'd be adding the network latency to each block check, which can become some significant time when checking millions of blocks, specially if your network equipment is not fast. Thus, checks should be performed on the backup server file system on NVME disks for best performance.

Let us pose some real life example:

In our lab, we have an old Synology box with regular Seagate HDs over gigabit LAN. One of the disks is mounted as an NFS3 volume. As you may imagine, the Synology CPU is quite outdated and extremely slow as per today's standards.

We have a Microtik switch in between the ESXi host and the old Synology box and the ESXi host is equipped with cheap Intel NICs

We check some 50 GB Windows 7 VM there from the Synology DSM command line and from the ESXi command line:

From ESXi:

-----------------------------------------------------------------------------------------------------------
     7/13                                    W702-flat.vmdk.map    53.00 GB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
     8/13                                      W702.vmx.tmp.map     3.30 KB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
     9/13                                         W702.vmsd.map    45.00 B    |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
    10/13                                          W702.vmx.map     3.30 KB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
    11/13                                         W702.vmdk.map   601.00 B    |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
    12/13                                        vmware.log.map   256.09 KB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
    13/13                                 W702.vmdk.extents.map     5.54 KB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-----------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-----------------------------------------------------------------------------------------------------------
real    3m 42.44s
user    0m 4.61s
sys     0m 0.00s

Now from the Synology command line:

-----------------------------------------------------------------------------------------------------------
     7/13                                    W702-flat.vmdk.map    53.00 GB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
     8/13                                      W702.vmx.tmp.map     3.30 KB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
     9/13                                         W702.vmsd.map    45.00 B    |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
    10/13                                          W702.vmx.map     3.30 KB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
    11/13                                         W702.vmdk.map   601.00 B    |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
    12/13                                        vmware.log.map   256.09 KB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------
    13/13                                 W702.vmdk.extents.map     5.54 KB   |  100.00%   100.00%        0   OK
-----------------------------------------------------------------------------------------------------------

real    0m51.451s
user    0m2.310s
sys     0m4.707s

The CPU capacity is not that important here when compared to the network latency added to the job. Even a crappy twelve year old CPU outperforms a fairly modern i5.

Each DC license comes with two keys, Pro just one. In case you need three licenses, you just have to buy one DC license plus one Pro license. Both editions are exactly the same software, DC comes with a discount though.

Please download latest version (we just released 1.4.3.3 today) and try again.
In case you still experience some issue, let us know what the issue is, provide as much details as possible, we can't guess it.

Some features may be pending to be added, some others may not fully work in Free version. Use some e-mail redirection instead.

You have mispelled the argument, it's [b]--backup-vms[/b] and not --backup-wms

Correct job:

./xsibackup --backup-type=custom --backup-vms="backup01" --backup-point=/vmfs/volumes/USB-Storage/CopiaSeguridad

Thank you very much for your feedback.
We will revise v. 1.4.3.1 and reach you back ASAP.

Update 2021-01-20 17:46 UTC:

There isn't anything drastically new from 1.4.3.0 to 1.4.3.1, but some little amendments and the automatic update of the server side binary in over IP operations.

Overlapping jobs could only trigger an error in the event that a block was being accessed at the same time, which is highly unlikely to happen. This is in turn prevented by a file lock mechanism.

Server side block processing is performed by spare subprocesses. We have tested this to some extent with no issue.

Slow disks would not cause any issue, just as long as they are O.K.

Do pay attention to the possibility that the NFS mount is not well mounted. Your last error does really look like if you were writing to the mount folder and not to the mounted volume, we see this very often.

We have to add some logic to check for real available space in the target folder. It seems too obvious to miss, but the above commented situation is extremely frequent.

Also, if your link or disk is not very reliable, do use Sync NFS instead of Async NFS.

It's much simpler than the load of questions you are posing yourself. The backup job is a file you save to the jobs folder, then you invoke it from the cron, that's it.

Everything is self contained in the job, you don't need to worry about any argument/ option but in the backup job file itself.

The VM name is the asigned VM Name as stated in the .vmx file displayName field, nothing else.

Yes, all backups are hot by default if the VM is on.

[url=https://33hops.com/xsibackup-cron-troubleshooting.html](c)XSIBackup Cron Troubleshooting[/url]

You aren't even posting the job that produced it.

Follow the Ockham's razor principle in this case too. If the output is telling you there isn't any VM to backup, then, the most likely cause is exactly that.

You might be making some wrong asumption, there may be something mispelled or your job might not be well built, but as you didn't publish it in the post, we can't offer you an answer.

New DC Free version works for some days, then it defaults to Free features: manual backup of VMs up to 60GB to local datastores.
Next version will also fully work for some days (10 days since the day of launch) until the trial period expires.

Well, it doesn't necessarily mean that you run out of space in the target volume, there are other volumes implied, like temp file systems or log space.

In any case if your repo grew over 1.5 TB you should upgrade to 1.4.3.0 that will fix it, if the problem was due to lack of resources.