Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

New (c)XSIBackup-Free does allow spaces in any part of the VM or virtual disks paths. Download our new [b][url=https://33hops.com/xsibackup-vmware-esxi-backup.html]VMWare Backup software[/url][/b] here

Right!
Anything regarding the old VMs is also something to be done, namely: turn them off and unregister them and in special removing the old cron jobs, as they could overwrite your new production VM with old data from the old VMs.

It should, just as long as your cond daemon is running OK.
verify with

ps -c | grep crond

There should only be one crond instance.

Jobs are named 000 to 999 as a fixed length three digit file name. If you use your own names you may hit some problem further on, as some features will expect jobs to be named that way.

Yes, job files are bash scripts, you can edit them and changes are immediate. If you have already scheduled your job file, there's nothing more to do. If you need your crontab to be persistent across reboots you must run.

./xsibackup --install-cron

And add your schedules to the local crontab at /vmfs/volumes/xsi-dir/conf/root-crontab, then run the --update-cron command

./xsibackup --update-cron

This will copy the contents of your local crontab to the (c)ESXi crontab at /var/spool/cron/crontabs/root.

We had posted a really long answer on this, but the problem has too many variables and too many things that we still don't know.

(c)ESXi's sort is not going to do it for such a big amount of data. Run the --repair command from the server side instead.
If you need some further help contact support directly to receive more personalized support.

Appliances such as (c)Synology and Qnap are great options when you need to provide some quick storage in the form of NFS or iSCSI, nonetheless they are closed down propietary environments. For what you want to do and the amount of data that you want to handle a full Linux OS with absolute control is a must.

The sort binary in Synology lacks the T option. You have a statically compiled sort binary in the /bin dir of the installation package that you can copy to your /usr/bin dir in the Synology appliance. Just keep a copy of the old one.

You don't need the xsilib binary in Linux, it is a helper only used when (c)XSIBackup is run in (c)ESXi. We don't know why it is being called in Rocky Linux, it could be a bug, we'll check that. In any case you can consider that strace output to be an spurious error. [b]Xsilib[/b] is used to traverse directory entries, it is not related to creating the .blocklog.tmp file.

If your --repair command is running fine and it's not returning any errors you can consider it to be running fine. As said, repair will just re-account the blocks in the restore points and rebuild the .blocklog file. We have designed all those files in text form so that you can actually see what is inside and how things work.

You could even use simple bash tools to partially rebuild the .blocklog file for some restore point only.

Is it 20.00TB of data on disk? or nominal data. If it is data on disk, you should be keeping your repos smaller to avoid resource usage growing wild when performing such actions as --repair or --prune. It is OK to just store backups, as the use of memory is extremely limited, in the order of some tens of megabytes.

What we mean is that --backup and --replica actions use a very limited amount of memory while --repair and --prune in special use a massive amout of memory.

You should be using a Linux host to accumulate that amount of data. (c)ESXi is not the adequate OS to host deduplicated data for many reasons, being the most important one that VMFS is not the right File System to store deduplicated repositories. We always take the chance to explain this in all related posts.

You need a fast file system and a real file system to store millions of chunks of data with maximum efficiency. We recommend that you use XFS or ext4 should XFS not be available. The Operating Systems that we recommend to store big repositories are CentOS 6/7 and Rocky Linux 8

Even if you use some NFS mount with some XFS or ext4 FS on the remote volume, trying to --repair from the (c)ESXi host is highly inefficient. Although NFS might look like a local file system to the eye of the user, it is not. Every system call you place on the remote storage will suffer from the corresponding network latency. This is OK when working with a limited set of files, like (c)ESXi does, nonetheless if will be extremely inefficient and unfeasible when working with big deduplicated repos.

On top of that the collection of utilities that come with (c)ESXi are a limited set of Busybox, they are in turn modified and stripped from some of their functions in some cases. Thus the sort binary that is throwing your -out of memory- error is not a full version of the well known Linux [b]sort[/b] utility.

(c)ESXi is not a full OS. It is a hypervisor. The use it makes of memory is not what you would expect from a full fledged OS. It will not use all available memory to run local binaries like [b]sort[/b]. This is not a problem to make your backups, still it will be when trying to perform operations that require massive amounts of CPU and memory. Thus, use the backup server to perform such actions instead of (c)ESXi.

As an excerpt:

- Host your deduplicated repositories in Linux. You may attach your linux box over NFS and perform local backups or send your backups over IP, that does not matter.
- Keep your backup repositories below a manageable size. This will be proportional to your hardware posibilities.
- Run your --repair and --prune operations from within the Linux backup server.

Apart from that there will still be a limit to what you can handle from within Linux, obviously the limit will be way higher than what you could achieve from within (c)ESXi over an NFS link. As (c)XSIBackup evolves it will become more powerful and refined, mostly by getting rid of the external binary dependencies and carrying out some actions by using more "built for the purpose" algorithms.

UPDATE:
Nonetheless, for the case being, --repair is a rather straight action. It collects blocks in the .map files on each restore point, loads them into a memory buffer, removes duplicates and checks for the physical blocks at the same time that it rewrites those blocks to the /data/.blocklog file or main block manifest file.

In your particular case --repair is failing in the first stages, while collecting blocks, it just can't grow the buffer any further, as the local shell is limited in resources when compared with what you want to achieve as previously explained. The solution is to run the very same command from the Linux backup host.

If you are hosting 20TB of data and, assuming that your compression ratio is just 90% (way below what (c)XSIBackup can achieve), you are managing 200TB of nominal data, which implies loading more than 5 billion block hashes into the memory buffer => 2.14 * 10^11 blocks. In your particular case you should devise a backup system, in special a backup host, that can really hold and work with that amount of data. That would require some dedicated M2 temp disk as cache in the (c)ESXi server (only if your VMs are really big) and a big amount of RAM in the backup host as well as a last generation CPU. The faster the disks are the better obviously, thus using M2 as the target of your backups would also be a great help.

Well, the e-mail subject is limited in size. We have in our to do list adding some kind of job variables that you can use in your job: job Id, session Id, VM list, etc... Still, that should probably go in the e-mail body where there aren't so much constraints in size.

It's not only the the subject field is limited in size but that it might be truncated or modified by some software like e-mail filters.

Anybody coming here because of the error:

| Error code 266 at file dedup-in.c, line 266 | Error             |
| description: can't open input file:  

It is due to (c)XSIBackup not being compatible with local datastore names containing spaces. We have released a fix in 1.5.1.12, just download and upgrade your installation.

Still, we encourage you to keep your datastore names short, concise and free of any special character.

(*) We deleted some previous entries in the post containing file listings, they are no longer needed and they expose some details on your setups.

Thank you.
We have conducted some tests using your very same (c)XSIBackup and (c)ESXi version in CentOS 7.0 as an IP backend with no issue.

We'll take the chance to revise the main steps involved in achieving your goal.

As said, using the root user is the recommended procedure, as it will abstract you from having to deal with the permissions of the elements involved in the process. On the other hand a public RSA key authentication over SSH is, by today's standards, the most secure way to access any remote OS.

You start by installing some fresh CentOS 7.0 OS and updating it via yum: [b]yum update -y[/b].

Now you can directly exchange your public (c)XSIBackup RSA key with the just installed host.

From the (c)ESXi box:

./xsibackup --add-key root@a.b.c.d:22

(*) Do make sure to enable SSH port in the (c)ESXi firewall, both in and out.

We start by exchanging the root key, as only the root user will be able to automatically update the remote binaries: xsibackup and base64
We then create the backup user in the remote CentOS 7 backend:

[root@localhost ~]# useradd backup1
[root@localhost ~]# passwd backup1
Changing password for user backup1.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

Create a dir to store your backups and take ownership of the home folder for this user:

mkdir -p /home/backup1/backups
chown -R backup1:backup1 /home/backup1

Add the (c)XSIBackup RSA public key to the [b]backup1[/b] user's authorized_keys file, namely: exchange the key again for this user.

From the (c)ESXi host:

./xsibackup --add-key backup1@a.b.c.d:22

The key contained within the [b]xsibackup_id_rsa.pub[/b] file in the (c)XSIBackup installation dir will be added to the remote OpenSSL file: /home/backup1/.ssh/authorized_keys, that is: the file containing the authorized keys for the [b]backup1[/b] user. You can also do that manually.

Now you should be able to access the remote system passwordlessly using the [b]backup1[/b] user. You can easily check that by running the last --add-key command again.

Still, you won't be able to backup to that host yet, as you still need to install the [b]xsibackup[/b] binary to act as the server side of your backups or replicas. Again, you can install manually or just use the built in automatic process.

To install the binaries to the remote host, just run some --backup or --replica to some folder on the remote CentOS 7 backup server by using the root user

./xsibackup --backup "VMs(YOUR-VM)" root@a.b.c.d:22:/home/backup1/backups

The required binaries will be updated and assigned 0755 permission.
Now run the same backup command again by using the [b]backup1[/b] user.

./xsibackup --backup "VMs(YOUR-VM)" backup1@a.b.c.d:22:/home/backup1/backups

We believe you already got to this point.

Now to restore any of the backup points for [b]YOUR-VM[/b] in the remote CentOS 7 server just create some local folder to host the restores.

mkdir -p /vmfs/volumes/datastore1/restores/YOUR-VM

And then restore the VM to the above generated folder.

./xsibackup --restore backup1@a.b.c.d:22:/home/backup1/backups/20220331122508/YOUR-VM /vm
fs/volumes/datastore1/restores/YOUR-VM

You will see some output like the one below

[root@localhost:/scratch/XSI/XSIBackup-DC] ./xsibackup --restore backup1@a.b.c.d:22:/home/backup1/backups/20220331122508/CRM /vm
fs/volumes/datastore1/restores/YOUR-VM
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
|||   (c)XSIBackup-Free 1.5.1.11: Backup & Replication Software                 |||
|||   (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved    |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
                   (c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
/!\ Running XSIBackup in a nested host is not supported
-------------------------------------------------------------------------------------------------------------
Remote system: Linux
-------------------------------------------------------------------------------------------------------------
PID: 2102784, Running job as: backup1
-------------------------------------------------------------------------------------------------------------
Run job as 'root' to automatically update the remote binaries
-------------------------------------------------------------------------------------------------------------
The restore directory: /vmfs/volumes/datastore1/restores/YOUR-VM does not exist
Do you wish to create it? (y/n): y

-------------------------------------------------------------------------------------------------------------
Restoring VM CRM from 192.168.3.221
-------------------------------------------------------------------------------------------------------------
NUMBER                                                                  FILE            SIZE       PROGRESS
-------------------------------------------------------------------------------------------------------------
1/20                                                               CRM.vmx.tmp       2.45 KB | Done   0.00%
-------------------------------------------------------------------------------------------------------------
2/20                                                             vmware-49.log     189.61 KB | Done   0.00%
-------------------------------------------------------------------------------------------------------------
3/20                                                                  CRM.vmdk     500.00 B  | Done   0.00%
-------------------------------------------------------------------------------------------------------------
4/20                                                             CRM-flat.vmdk      20.00 GB | Done  99.99%
-------------------------------------------------------------------------------------------------------------
5/20                                                               CRM-aux.xml      13.00 B  | Done  99.99%
-------------------------------------------------------------------------------------------------------------
6/20                                                          CRM.vmdk.extents       8.30 KB | Done  99.99%
-------------------------------------------------------------------------------------------------------------
7/20                                                                  CRM.vmxf     258.00 B  | Done  99.99%
-------------------------------------------------------------------------------------------------------------
8/20                                                                 CRM.nvram       8.48 KB | Done  99.99%
-------------------------------------------------------------------------------------------------------------
9/20                                                                  CRM.vmsd      46.00 B  | Done  99.99%
-------------------------------------------------------------------------------------------------------------
10/20                                                            vmware-51.log     161.16 KB | Done  99.99%
-------------------------------------------------------------------------------------------------------------
11/20                                                     CRM-flat.vmdk___hash      82.00 B  | Done  99.99%
-------------------------------------------------------------------------------------------------------------
12/20                                                                  CRM.vmx       2.45 KB | Done  99.99%
-------------------------------------------------------------------------------------------------------------
13/20                                                               vmware.log     241.14 KB | Done  99.99%
-------------------------------------------------------------------------------------------------------------
14/20                                                            vmware-47.log     374.03 KB | Done  99.99%
-------------------------------------------------------------------------------------------------------------
15/20                                                            vmware-48.log     325.92 KB | Done  99.99%
-------------------------------------------------------------------------------------------------------------
16/20                                                             CRM.vmsd.tmp      46.00 B  | Done  99.99%
-------------------------------------------------------------------------------------------------------------
17/20                                                            vmware-46.log     172.79 KB | Done  99.99%
-------------------------------------------------------------------------------------------------------------
18/20                                                            vmware-50.log       1.45 MB | Done 100.00%
-------------------------------------------------------------------------------------------------------------
19/20                                                                 CRM.vmx~       2.45 KB | Done 100.00%
-------------------------------------------------------------------------------------------------------------
20/20                                                    CRM-Snapshot1470.vmsn      27.37 KB | Done 100.00%
-------------------------------------------------------------------------------------------------------------
Restore end date: 2022-03-31 12:52:38
-------------------------------------------------------------------------------------------------------------
Time taken:     00:08:11 (491 sec.)
-------------------------------------------------------------------------------------------------------------
Speed:           41.72 mb/s
-------------------------------------------------------------------------------------------------------------
Removed host <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed prog <tmp> dir        OK
-------------------------------------------------------------------------------------------------------------
Removed PID                   OK

To be able to restore your VM you just need the [b]backup1[/b] user have:

1/ Execute rights on the remote binaries. This is assigned by default when you use the root user to perform some test backup.
2/ Read access to the backup respository.

Of course, we are posing this example asuming that you are logged in the (c)ESXi server as root. It's not the same the local user than the remote.

To detect any possible issue in the SSH tunnel, you can debug it by adding the --debug-print argument to your restore backup, which will print the details on the remote commands launched:

./xsibackup --restore backup1@a.b.c.d:22:/home/backup1/backups/20220331122508/YOUR-VM /vm
fs/volumes/datastore1/restores/YOUR-VM  --debug-print

You can also run some SSH command to verify that you can access the remote files

From the [b]/scratch/XSI/XSIBackup-DC[/b] dir:

ssh -i xsibackup_id_rsa backup1@a.b.c.d "cat /home/backup1/backups/20220331122508/YOUR-VM/YOUR-VM.vmx.map" | cat > /tmp/YOUR_VM.vmx.map

This command should bring your remote [b]YOUR-VM.vmx.map[/b] file to the /tmp dir. It should contain a 40 byte SHA-1 hash.

To increase verbosity in the SSH tunnel and get all the details, in case you are getting some error:

ssh -vvv -i xsibackup_id_rsa backup1@a.b.c.d "cat /home/backup1/backups/20220331122508/YOUR-VM/YOUR-VM.vmx.map" | cat > /tmp/YOUR_VM.vmx.map

The job is running OK, it's the final part where the VM is registered where you are having some issue.
Maybe some duplicate name in the registry?

You don't need [b]--backup-how=hot[/b], it's the default behavior.

We need some more verbosity, run the job in the command line to debug it and add [b]--debug-print[/b] at the end to have the details on the command that fails.

./xsibackup --replica=cbt "VMs(vm)" "/backup" --options=R --quiesce --config-backup --debug-print

Update:
Looks like the problem is related to creating the remote replica snapshot. This is required to be able to switch the remote VM on without affecting the remote clone consistency and should never be deleted. It is in fact called "xsirep_donotdelete".

Your simplest workaround consists in unregistering the _XSIREP VM and deleting the clone folder from disk, this will recreate the remote cloned VM from scratch copying all data again.

You can't effectively use our software by just running the GUI. The GUI is a helper layer for productivity, it doesn't pretend that you abstract yourself from knowing how to deal with the command line.

In your case, your (c)ESXi version is lacking a date command modifier and you need to tweak that manually.