Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

Are you using latest version?.
Restore using [b]root[/b] and see if that works.

We recommend that you use CentOS 6/7 or Rocky Linux 8 as your target Linux system. There are hundreds of Linux distros and although most of them share some common principles, we can't guarantee that.

If you use the recommended distros and File Systems (XFS, ext4), you will enjoy the benefit of using some heavy tested scenarios and you will minimize your issues and take maximum advantage of your time.

Your problem broken down into its constituent pieces:

To be able to restore from a remote system you need xsibackup binary present in it. (c)XSIBackup can automatically update your binaries in the remote host, just as long as you use the root user, as the binaries are placed in the /usr/bin folder and you need elevated privileges to do that. You can also install manually but doing it through the regular over IP backup/replica command is much more convenient.

Once the binary is present in the remote system you will be able to use it when running the command with some alternative user, like [b]xsibackup[/b]. Still that user will need to have privileges to create folders inside its home directory /home/xsibackup. You can achieve that by running.

chown -R xsibackup:xsibackup /home/xsibackup/

You have obviously configured your backup server up to this point, as you say that you can indeed backup without issue using the [b]xsibackup[/b] user.

Restoring from the remote folder using the non-privileged user should be rather straight from this point. Just make sure that you are using the latest binary.

Thank you, your code snippet will for sure be useful to others. Just reminding you all, use with caution and test thoroughly.

CBT support is exclusive to DC version, still regular differential replica will be almost as fast for VMs under 50 GB.

Remember to use [b]xsibackup --prune /path/to/repository/20220325002123[/b] to remove restore points inside a repository. You can on the other side remove replicas by just removing the folder. Some people like to [url=https://33hops.com/xsibackup-rotating-replicas.html]rotate small groups of replicas[/url] when the total size of their VM set is not very big. If you add --replica=cbt, the update of each replica will be almost instant from the second run in each replica point.

(c)XSIBackup Classic will always create thin or sparse disks, if the underlying FS supports this type of file. It wouldn't make sense to do it other way, as you can easily turn any thin disk into a thick provissioned one and you want a backup program to take advantage of space as much as possible.

Some [b]date[/b] binaries in some (c)ESXi builds lack the [b]%j[/b] (Julian day) modifier (why is a conumdrum to us). Try this instead:

./xsibackup --replica=cbt "VMs(vmname)" "root@192.168.1.1:22:/vmdks/replicas/$(( $(date +%s)/86400%5 ))"

It just doesn't work that way.
We do not state anywhere that the software works that way (pulling VMs from one remote server to the local one). It works right the opposite way, you push VMs from the server where they are to a target server.

Please read the manual and examples, run some simple jobs to try the software out and then add more arguments as you learn to use the software.

Using (c)XSIBackup is very straight when you keep things under some fair degree of control. That means migrating VMs between hosts that keep some minimum level of uniformity: compatible SSH versions, compatible hardware versions and also compatible VMFS versions.

There are many different builds even under the same version number, not only those released by (c)VMWare but also from many manufacturers: DELL, HP, Fujitsu, etc...

Pushing VMs from 5.5 to 7.0 should be straight most of the times, still if your environment requires it, you may need to enable some deprecated cipher in the sshd_config file of the target system, disable FIPS, open ports in between the servers, etc.... There is no way you can escape from learning how OpenSSH works once you cross some given threshold of complexity.

First of all you have to take some things into account:

Make sure that you have indeed enabled the SSHOut rule in your (c)ESXi firewall, not only SSHIn.

The OpenSSH versions in your two (c)ESXi boxes differ too much.
As new OpenSSH versions are released, the KEX algorithms and Ciphers are deprecated and new ones are added. Thus, it's likely that two OpenSSH implementations that are too distant in time can't agree on some KEX or Cipher.
You can try to solve the above mentioned situation by re-enabling some deprecated algorithm in your newest (c)ESXi box, should that be possible, as (c)ESXi's OpenSSH implementations are customized and may not allow to re-enable some older algorithms, contrary to regular OpenSSH releases.

Nonetheless, in your case you aren't even getting to that point. [b]Your problem is related to having FIPS active in the (c)ESXi 7 box while your 5.5 box does not even support this feature[/b]. This is a common error when connecting from FIPS to non FIPS OpenSSH servers.

Now the thing is: how to disable FIPS in the target server. Is that even possible?

In (c)ESXi 6.7.0 and above:

1/ To find whether FIPS is enabled or not.

esxcli system security fips140 ssh get

2/ To disable FIPS

esxcli system security fips140 ssh set --enable=false

(*)FIPS is a security feature. You may very well disable it to facilitate a migration, you should not disable it permanently though.

Migrating VMs from 5.5 => 7.0 is possible by using the workaround above, although you may need to tweak some other things depending on your exact (c)ESXi version and build.

Doing it the other way around 7.0 => 5.5 could be a real pain if possible at all.

In your post you say that you are migrating VMs 5.5 => 7.0, you then talk about not being able to connect from 7.0 => 5.5. Is that an error in your description?, or there is something we are missing.

UPDATE:
New (c)XSIBackup-App appliance runs on CentOS 7 which is a full fledged Linux OS. Thus it will allow you enable any deprecated algorithm in OpenSSH as well as install any software that you may need. It's a much more flexible tool, as it doesn't run in the (c)ESXi OS and is not constrained by its limitations and cut down features. Download at Sourceforge.net: [url]https://sourceforge.net/projects/xsibackup-app/[/url]

Yes it does

Yes, you can replicate anywhere you want, since you can replicate to any remote FS over IP/SSH.

Please, let us politely insist that what you are planning to do makes no sense at all from a disaster recovery perspective. Independently of whether you will switch the VM on or not, that is irrelevant. In fact it is very recommendable to switch the remote VM on from time to time to make sure that your replica is indeed working, our replica algorithm allows you to do so while still preserving the replica consistency.

Any problem in the original VM will be almost immediately transmitted to your replica. Let's say that you are affected by some ransomware?, what would you do to recover your data?

With the Classic version it was a different thing, as it used a bigger block size.
There are two main concerns when using some FS to store deduplicated content in the FS as a database:

- The FS must be fast
- The FS must be able to handle the data

VMFS is slow, as it was not designed to handle millions of small files, but on the contrary to handle a few big files: the virtual disks.

VMFS is terribly slow when it comes to handle many small files, although VMFS-6 is indeed able to handle millions of files, it is very slow. On the other hand, apart from also being slow, VMFS-5 has an inode limit of aproximately 130.000 files and folders. When it reaches that limit it just blows up.

When using (c)XSIBackup Classic to accumulate data in old XSITools 1.0 repositories all these problems are minimized: first of all cause you are using a big block size (50MB), which reduces the effect of data access slowness by a factor of 50. Also the VMFS-5 inode limit is relative, as 100.000 inodes can store 5.0TB of real data, which may in turn host 15-30 TB in VMs.

(c)XSIBackup-DC is more optimized for speed and data compression. Classic version is a toy when compared to it. It uses a 1MB default block size, which you shouldn't change and which yields awesome compression figures, apart from offering CBT compatibility and a lot of extra features when compared to Classic version.

Still, it uses a block size which is 50 times smaller and this will make VMFS slow behaviour to become obvious and be much slower than expected processing data. In regards to VMFS-5, don't even try to use it for obvious reasons. Just 100GB of real data will make it stop responding.

Thank you for your feedback.

This is yet another issue having to do with quiescing your FS. We are writing about this all the time, still we have recently updated the main post relative to this topic and [url=https://33hops.com/esxi-snapshot-errors-and-solutions.html#quiescing-notes]added some specific notes[/url].

Every user should try to make the effort to see this kind of problems as a broad issue, even though each particular situation should require a slightly different procedure to solve it.

Of course your proposed solution will always work, as you are shutting your server down before taking the backup snapshot. Even though it is immediately switched on after taking it, the snapshot is indeed taken from a stopped state of the VM, thus the possibilities that your Active Directory DB gets corrupted are zero.

If you can afford to stop the VM for some seconds, a warm backup is definitely the simplest solution to this kind of problems. Still, not everybody can afford to stop the DC to backup the AD VMs.

[b]Problem description[/b]

AD information is kept in a DB. That DB could become corrupt, just like any other DB server which is abruptly stopped. The snapshot issue is about the same as a sudden power outage, which before virtualization became popular was the most frequent way to corrupt the AD database.

The mere fact that it does indeed become corrupt is random and proportional to how busy it is. You might be lucky and the service might be iddle just when you take your snapshot, you should not count on that though.

The DB becoming corrupt does not mean that the whole database goes corrupt. People tend to think in maximalistic terms all the time, which causes terror, doubt and in the end wrong decisions.

Databases become corrupt on power outages or non-quiesced snapshots just because the last pages that are being written get chopped before the end of the page is written to disk. Thus, the system preprocessing routines detect this unfinished write because some page in the DB lacks a footer or closing structure.

Fixing the problem consists in the same conceptual thing in every case: detecting the wrong pages and removing them, which is usually done with the database repair commands. This obviously varies depending on the DB system. In case of a DB server like MySQL or MS SQL Server, you would just loose the last writes or updates. In case of AD, the repairing would chop off the latest AD related operations.

Active Directory adds an additional problem, which is that the DC controller is dependent on the healthyness of the Active Directory DB to boot up. This could be considered an OS design flaw, as it puts you in a technical paradox. The solutions proposed by Microsoft don't seem to work in your case, still, there should be a fairly easy way to fix that DB, as said, this is an old issue which has mature fixing procedures since many years ago, as stated, power outages were a common source of AD relates corruption problems before they were replaced in frequency by virtualization snapshots.

[b]Quiescing backups[/b]

All this kind of issues are prevented the same way: quiescing the FS before actually taking the snapshot. It consists in about the same as a controlled shutdown for DB services, still done with the OS running and resuming normal operations ASAP. It usually takes some seconds at most to quiesce the different DB services in a server.

In the [url=https://33hops.com/esxi-snapshot-errors-and-solutions.html#quiescing-notes]notes on quiescing[/url] we describe the procedure to follow in case of DB services in Windows servers.

There are a few services related to quiescing a Windows guest: VSS, VMWare Tools, Virtual Disk and in some cases some additional helper services. Just as long as those services are configured as described in our post and all other related services are installed and configured properly, using a quiesced snapshot should prevent any corruption on the different DB services that may be running in your guest.

Quiescing in a nutshell consists in the (c)ESXi server communicating a snaphot is about to be taken to the VMWare Tools service in the guest, then the VMWare Service should coordinate the controlled pause of the running DB services.

Still, if you have some host that is not responding to automatic quiescing. You can control the process on your own, how?:

(c)VMWare Tools offer a way to run custom pre and post backup scripts, like described in the post. This scripts can handle three events related to snapshots: pre-FREEZE, THAW and FREEZEFAIL.

FREEZE happens right before the snapshot is taken, THAW happens right after the snapshot has been created (please, note that some documents on the web wrongly describe THAW as happening when the snapshot is deleted), finally FREEZEFAIL is run in the event that some error is triggered.

Controlling your AD services quiescing on your own would consist in adding the necessary AD Service stop command to FREEZE and AD Service start command to THAW, as well as to FREEZEFAIL. That way you make sure that before your backup snapshot is taken the AD Service is stopped gracefully preventing any data corruption and that once the snapshot has been completed it is started again.

It is conceptually the same as running a "warm" backups, still, you make sure that you don't have to reboot the server. It is indeed the same that the coordinated services in the server should do when they are configured the right way.

@echo off
if "%~1" == "" goto USAGE
if %1 == freeze goto FREEZE
if %1 == freezeFail goto FREEZEFAIL
if %1 == thaw goto THAW
:USAGE
echo "Usage: %~nx0 [ freeze | freezeFail | thaw ]"
goto END
:FREEZE
net stop YOUR_AD_INSTANCE_NAME
goto END
:FREEZEFAIL
net start YOUR_AD_INSTANCE_NAME
goto END
:THAW
net start YOUR_AD_INSTANCE_NAME
goto END
:END