©XSIBackup-Free: Free Backup Software for ©VMWare ©ESXi

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software


You are not logged in.

#26 Re: General matters » SMTP not working on 2.1.0.0.009 » 2023-11-16 12:10:53

It's already ready to download:
[url]https://sourceforge.net/projects/xsibackup-app/files/[/url]

#27 Re: General matters » When trying to do a delegate backup I get the following error » 2023-11-16 12:10:12

Run adding --debug-print to see why that command is failing.

#28 Re: General matters » When trying to do a delegate backup I get the following error » 2023-11-15 10:04:54

IS your ©ESXi host (192.168.123.11) version 8?.
The execInstalledOnly directive exists only in ©ESXi 8.
Apart from that running a delegated job requires a scratch partition configured in every ©ESXi server.

#29 Re: General matters » SMTP not working on 2.1.0.0.009 » 2023-11-15 10:00:35

In the ©XSIBackup-App appliance we are using [url=https://www.rapid7.com/products/insightvm/download/thank-you/?utm_medium=email&utm_source=email&CS=email]Swaks[/url] as an SMTP client by now, which requires authentication.

This will change in next version 1.1.0.1, the new default SMTP client will be cURL in sake of compatibility and to limit space usage, as Swaks depends on Perl and in the end requires more than 45 MB for just an e-mail client.

©XSIBackup-App 1.1.0.1 will also allow to select the classic ©XSIBackup sendmail module, which has been ported to Linux, via the --smtp-client=sendmail argument. Sendmail module uses STARTTLS (port 587) while cURL method (default) uses a direct TLS/SSL connection on port 465.

All these changes are transparent to the user 99% of the times, except in cases like yours in which you are using an open relay and Swaks does not support that configuration.

#30 Re: Feature requests & improvements » automated screenshot of replicas » 2023-11-15 09:52:44

We moved this to Feature Requests & Inprovements.
We will assess the possibility to add such feature, still a screenshot of a running VM means nothing.
The ideal certification would be to compare the virtual disks' hash to the replicated VM disks' hash. This is most probably the way we will go to offer some kind of additional backup certification. Take on account that we are already checking the number of bytes sent and received and every individual checksum for each 1 MB block.

#31 Re: General matters » Incomplete Restoration and Mounting Error with XSIBackup on ESXi 7.0 » 2023-11-10 14:57:40

You have posed your question in the [b]©XSIBackup Classic Free[/b] forum. Is this the version you are using?.
Please clarify to move your question to the right forum if needed.

You seem to simply not have formatted the partition you are trying to mount.
[url]https://unix.stackexchange.com/questions/315063/mount-wrong-fs-type-bad-option-bad-superblock[/url]

#32 Re: General matters » backup a domain controller » 2023-11-08 11:29:21

Yes, of course, that should always work if the VM is off.
The key matter here is that when you use --backup-how=warm|cold ©XSIBackup does issue a controlled shut down, thus this technique must always work, unless...: the configured timeout for giving up a controlled shut down and perform a simple power off in the xsibackup.conf file is not enough.

If it is the power off that is bringing the VM down, then you may end up with a corrupted AD.

#33 Re: General matters » backup a domain controller » 2023-10-25 20:22:58

Thank you for your feedback. There is something going on there. From time to time some user tells us about something similar.

The thing is that a correct shut down or snapshot with proper service management should yield a fully working system. If you get a BSOD and you do know that the BSOD is related to AD, then there is only one possible cause: [b]the AD service wasn't properly shut down[/b]. Your BSOD is clearly stating that point with the [b]0xc00002e2[/b] error (this error is an indication that the Active Directory database NTDS.DIT is corrupt).

Thus, the solution is conceptually simple, yet, the usual MS fuzz around simple issues seems to reach it's peak around this problem, which has been around for decades. Virtualizing Windows Server only makes the problem worse, but this already happenned in hardware based servers 20 years ago. The simple solution to this is: [b]how the H. to shut the AD service down in a controlled way[/b].

It is rather obvious that a normal Windows shut down should yield a working system, thus using the --backup-how=cold argument has to work. There is only one thing to take on account in that case and it is to make sure that ©XSIBackup was indeed able to perform a controlled shut down and not a power off instead. Increase the timeout in xsibackup.conf to make sure this is indeed what's happening. The event viewer in the Windows OS should make it clear whether the previous shutdown was performed correctly or it was a mere power off.

--backup-how=cold does shut the server down and then backs up the VM from a stopped state, this must yield a fully working system 100% of the times, just as long as the shut down was indeed controlled. Please do make sure that your VM is not being powered off instead, we are sure this is the only possibility in case you are still getting the BSOD after a cold backup or replica.

UPDATE:

The command

net stop NTDS /y

Should produce a controlled shutdown of the AD services. If you are getting a BSOD after running it, then your AD service is not working right, you may need some additional patch for your OS or some additional service installed.

We would start by issuing [b]net stop NTDS /y[/b] manually, then reboot. Also trying [b]net stop NTDS /y[/b] followed by [b]net start NTDS[/b] and making sure that AD is still working right would help determine whether the issue is with the AD service itself or with the VMWare Reboot process.

Also make sure that all services related to snapshots are present in the way described in our [url=https://33hops.com/troubleshooting-windows-snapshots-in-esxi.html]post on Windows snapshots[/url]:

Volume Shadow Copy
Microsoft Software Shadow Copy Provider
VMware Snapshot Provider
COM+ System Application
COM+ Event System
Virtual Disk

Things such as having Secure Boot features enabled in your BIOS can also contribute to create these type of issues.

As a workaround we suggest that you backup the C:\windows\ntds\ntds.dit file from a known working state to be able to easily restore it in case of need.

#34 Re: General matters » backup a domain controller » 2023-10-23 09:21:36

Yes, AD is a complex service it will depend on other services.
We assume that you did use a freeze script to issue the [b]net stop NTDS /y[/b] command, you don't mention.
Did you check in the Event Viewer that the freeze script was actually run?.

Before trying the quiesce from VMWare Tools do try the process manually:

1/ Issue the [b]net stop NTDS /y[/b] command.
2/ Power off the VM. Do issue a hard power off, so that if AD is still active the power off would mimick a plain snapshot.
3/ Turn on the VM and check the state of the AD service. The AD service should start normally, the same way as after a controlled shut down.

If the above succeeds.

1/ Issue the [b]net stop NTDS /y[/b] command.
2/ Backup the VM with a plain snapshot, no quiescing.
3/ Turn on the VM and check the state of the AD service. The AD service should start normally, the same way as after a controlled shut down.

If the above succeeds then the AD service is being properly stopped. If it still fails, then it's the [b]net stop NTDS /y[/b] command that isn't fully working.

#35 Re: General matters » backup a domain controller » 2023-10-20 08:24:04

Thanks for the link. We published a post with a more [b][url=https://33hops.com/esxi-snapshot-errors-and-solutions.html]comprehensive approach to freeze scripts[/url][/b] some time ago.

It's somewhat awkward that MS has not published detail information on how to stop AD services in a controlled manner. There is very little information on the subject. Nonetheless AD is a complex service dependent on a database where it stores its configuration. The principles that apply are the same as with any other system dependent on a consistent I/O scheme.

Just as long as you are able to stop it in a controlled way you got it. Controlled freezing (stop service before taking the snapshot and restart after the snapshot was done) is even better, as you don't need to stop the VM.

#36 Re: General matters » Segmentation fault when restoring over IP » 2023-10-19 20:27:39

We have been running a backup job similar to yours for some hours now.

n=0;while [ "$n" -lt "100" ];do xsibackup --backup "VMs(ALL)" "root@a.b.c.d:22:/home/backup/repo02-1MB" --rotate="4";n=$(( n+1 ));done

We have also been banging the two guests in the ©ESXi lab host with big downloads and then removal of the downloaded files to make sure that the number of blocks would be changing all the time. No quiescing has been performed, thus some files have probably been backed up partially in some of the backup rounds.

We haven't been able to reproduce your issue so far. As commented before we maintain several backup schemes at clients with prunes and --rotate commands that, as you probably already know, is nothing but the pruning of the eldest restore points.

We can offer you some ideas on what might be going on:

1/ Broken disk? just thinking loud.
2/ Broken FS.
3/ Accidental deletion of blocks.
4/ Attempt to restore from some leftover restore point corresponding to some pruned dir.
5/ Broken restore points due to backup tests or failed backups. This would not affect other restore points corresponding to successful backups though.

If you can't find the cause contact support to provide more details.

#37 Re: General matters » backup a domain controller » 2023-10-19 19:04:52

Sorry for reaching back so late, your last response passed inadvertent to us.

AD machines are picky to quiesce, probably pickier than they should be. The AD database should be more resilient to blackouts and simple power offs, anyway...

Both warm and cold --backup-how methods should work with any system. The reason is simple: a "controlled shutdown" (we'll get back to this in the next paragrah) precedes the snapshot, thus the snapshot is taken from a stopped state of the VM. That stopped state of the VM should yield a perfectly coherent state of the VM as the shutdown process must take care to bring down all services in a controlled maner including AD services. This controlled shut down should take care to flush all pending data to disk and make sure that the AD DB is in a perfectly coherent state, just as if you were switching off any hardware based server.

Now the thing is whether the shutdown was really a controlled shutdown. ©XSIBackup tries to perform a normal controlled shutdown and if it can't it issues a power off. You can find a tweakable variable in the [b]etc/xsibackup.conf[/b] file to control the time ©XSIBackup waits before issuing a plain power off.

If your server is busy and it takes longer than the default configured time ©XSIBackup may be powering off your AD server instead of shutting it down gracefully causing the AD database to become corrupt. Check that in the backup log and extend the default timeout if you need to.

You can also use a freeze script via VMWare Tools (see previous post above) to switch AD services off before the snapshot is taken, this would allow you to backup the VM while it is on.

The key is to make sure the AD related services are brought down in a controlled manner to avoid corrupting the AD database.

#38 Re: General matters » Segmentation fault when restoring over IP » 2023-10-19 17:42:01

We have repos in clients that have been rotating for years and all non pruned blocks are there.
The most obvious explanation is that you may be trying to restore some restore point that was indeed pruned but for some reason the .map files weren't deleted at the end of the prune process.
Please contact our support department with as much details as possible, we will check the exact same kind of job in our lab in the meanwhile.

#39 Re: General matters » Backup VMs on a VSAN Datastore to NFS using ©XSIBackup? » 2023-09-28 10:56:45

(c)XSIBackup IS NOT compatible with vSAN.
Any attempt to backup VMs in a vSAN volume will yield undefined results.

#40 Re: General matters » XSIBackup Free 100 GB limit error for 50 GB VM » 2023-09-28 10:55:43

It is indeed the nominal size of the disks that is taken into account.

#41 (c)XSIBackup-Pro & DC bug tracker » CBT wrong info in -ctk.vmdk files after reboot » 2023-08-28 11:55:32

admin
Replies: 0

Some ©ESXi versions reset cbt after a hard reboot. This forces to reset the CBT configuration to prevent loosing data or corrupting your backups. We have added some logic to handle this kind of situation.

#42 Re: General matters » Error code 222 at file signal.c, line 222 | Error description: Signal » 2023-08-21 13:30:19

Your build is not from ©VMWare. Our software is only tested with official ©VMWare builds. As per our own experience manufacturer builds may contain arbitrary changes probably due to different reasons.

In any case, please post the job as you used it along with the output, otherwise we can do nothing but to especulate. SIGNAL 11 is a generic exception with no particular meaning.

To save time add [b]--debug-print --test-mode[/b] to the end of your job and post the output.

#43 Re: General matters » Segmentation fault when restoring over IP » 2023-08-15 21:24:26

If you have missing blocks you can't create them out of thin air. You should check your repos from time to time to prevent this kind of thing from happening. We can't assess what is the reason for the block loss, it could be from a damaged disk to incomplete backups that returned some errors or a partially deleted repository.

First of all you should use some newer version to restore. Restore functionality is not limited by any license check, thus you can use the free version to restore.

If you use the --force argument at the end of your restore command you will be able to restore your disk without the missing blocks, just holes instead of those blocks.

If you keep getting a segfault, then the repo could be broken. The reason for that could be missing or damaged files or them having been edited with some text editor that changed the line ends or the file encoding.

You should check that you have an .xsitools file in the root of the repo that you can read with cat or vi, a data folder with a block manifest named .blocklog and the backup folders in the repo root with .map files that contain the constituent blocks.

.xsitools file

Desc: XSITools Repo v 2.0.0
Bsiz: 1048576
Comp: 1

The data folder contains a hierachical structure with the block data in a six level nested folder set.

[root@localhost ~]# ls -la /home/backup/repositories/repo01/data/
total 360
drwxr-xr-x. 18 root root    167 Aug 15 22:51 .
drwxr-xr-x.  8 root root    145 Aug 15 22:51 ..
drwxr-x---. 18 root root    150 Aug 15 18:51 0
drwxr-x---. 18 root root    150 Aug 15 18:52 1
drwxr-x---. 18 root root    150 Aug 15 18:51 2
drwxr-x---. 18 root root    150 Aug 15 18:51 3
drwxr-x---. 18 root root    150 Aug 15 18:51 4
drwxr-x---. 18 root root    150 Aug 15 18:51 5
drwxr-x---. 18 root root    150 Aug 15 18:51 6
drwxr-x---. 18 root root    150 Aug 15 18:51 7
drwxr-x---. 18 root root    150 Aug 15 18:51 8
drwxr-x---. 18 root root    150 Aug 15 18:52 9
drwxr-x---. 18 root root    150 Aug 15 18:52 a
drwxr-x---. 18 root root    150 Aug 15 18:52 b
-rw-r--r--.  1 root root 366121 Aug 15 22:51 .blocklog
drwxr-x---. 18 root root    150 Aug 15 18:52 c
drwxr-x---. 18 root root    150 Aug 15 18:51 d
drwxr-x---. 18 root root    150 Aug 15 18:51 e
drwxr-x---. 18 root root    150 Aug 15 18:52 f

Some repo restore point folder:

[root@localhost ~]# ls -la /home/backup/repositories/repo01/20230815227220/WXPMK
total 1048
drwxr-xr-x. 2 root root    4096 Aug 15 22:51 .
drwxr-xr-x. 3 root root      19 Aug 15 22:51 ..
-rw-r--r--. 1 root root      97 Aug 15 22:51 vmware-10.log.map
-rw-r--r--. 1 root root      97 Aug 15 22:51 vmware-11.log.map
-rw-r--r--. 1 root root     146 Aug 15 22:51 vmware-12.log.map
-rw-r--r--. 1 root root      48 Aug 15 22:51 vmware-13.log.map
-rw-r--r--. 1 root root      97 Aug 15 22:51 vmware-14.log.map
-rw-r--r--. 1 root root      48 Aug 15 22:51 vmware-15.log.map
-rw-r--r--. 1 root root      48 Aug 15 22:51 vmware.log.map
-rw-r--r--. 1 root root 1003520 Aug 15 22:51 WXPMK-flat.vmdk.map
-rw-r--r--. 1 root root     118 Aug 15 22:51 WXPMK.info
-rw-r--r--. 1 root root      45 Aug 15 22:51 WXPMK.info.map
-rw-r--r--. 1 root root      46 Aug 15 22:51 WXPMK.nvram.map
-rw-r--r--. 1 root root      45 Aug 15 22:51 WXPMK.vmdk.map
-rw-r--r--. 1 root root      44 Aug 15 22:51 WXPMK.vmsd.map
-rw-r--r--. 1 root root      44 Aug 15 22:51 WXPMK.vmsd.tmp.map
-rw-r--r--. 1 root root      46 Aug 15 22:51 WXPMK.vmx.map
-rw-r--r--. 1 root root      46 Aug 15 22:51 WXPMK.vmx~.map
-rw-r--r--. 1 root root      46 Aug 15 22:51 WXPMK.vmx.tmp.map

Detail of some .map file

[root@localhost ~]# tail -n20 /home/backup/repositories/repo01/20230815227220/WXPMK/WXPMK-flat.vmdk.map
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
6bf4834b3c2b6d46e03886a99b31e75b5dc6df00;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576

The end of this -flat.vmdk file contains mostly 1MB zeroed blocks (3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3), thence the hash repetition.

#44 Re: Repositories » Repo on VMFS6 and block size related to file system capacity. » 2023-08-15 21:09:09

It is encouraged all over our documentation not to use VMFS as the target for deduplicated backups but some XFS or ext4 file system instead.

You don't need 20 cores nor 320GB of RAM. If you use 20 cores 18 will be iddle. (c)XSIBackup uses two cores. You should better use some low number of cores powerful CPUs, such as some double core Intel Gen above 10th than a CPU with a lot of weak cores such as AMD Epyc series.

In regards to the RAM if you add 320GB you will be wasting 318GB, unless you need to do some really intensive pruning.

#45 Re: General matters » Disk on different datastore getting excluded due to naming conflict? » 2023-08-05 10:50:05

(c)XSIBackup-App detects duplicate disks and prepends the datastore name to the disk file when using --backup action. This feature will be available to (c)XSIBackup-DC in next version.

You should name disks uniquely within one VM though. (c)XSIBackup consolidates all disks in one folder in order to produce a working VM when restoring. It does its best at the time to rearrange disks in a new working .vmx file, still that is just a recomposition of the original set up. We can't assume that (c)XSIBackup will find the exact same infrastructure when restoring, as this would be counterproductive in most DR situations, thus we always aim at producing something that will work from a single directory.

When you restore such a VM you will end up having to rearrange things according to the original layout or to a totally new reality. Naming disks uniquely will guarantee that the restored VM will work out of the box without having to eventually edit the .vmx file.

You should rename disks manually to be 100% sure on what is the final result as the VSphere GUI is just an abstraction layer and doesn't always do what you would expect.
Renaming a disk is rather simple:

1/ Stop the VM.
2/ Change the -flat.vmdk file name.
3/ Edit the .vmdk file descriptor to point to the new -flat.vmdk file and remove any CBT related lines. The [b]# Change Tracking File[/b] section.

# Change Tracking File
changeTrackPath="VM01-ctk.vmdk"

4/ Edit the .vmx file to point to the new .vmdk file descriptor and delete any CBT related lines, usually at the end of the file if enabled through --enable-cbt argument.

ctkEnabled = "TRUE"
scsi0:0.ctkEnabled = "TRUE"
scsi0:1.ctkEnabled = "TRUE"
scsi0:2.ctkEnabled = "TRUE"

5/ Reboot the VM and eventually run --enable-cbt again.

UPDATE:

There is already a version available (1.7.0.7) which addresses duplicate named disks by prepending the datastore name to them. Please note that when you restore the VM you will have to edit the .vmx file after deciding where to put those disks.

#46 Re: General matters » Segmentation fault when restoring over IP » 2023-08-03 09:07:48

You could have run out of TMP space. As your version is a bit outdated we would update or at least try the restore with a more recent version.

To see the details on what is making the restore process to SEGFAULT prepend strace to your restore command.

strace ./xsibackup --restore root@backupserver:22:/nas/xsibackup/weekly/20230723150052/VM1/ /vmfs/volumes/datastore1/RESTORE

To check your backup at the backup server run this command.

xsibackup --check /nas/xsibackup/weekly/20230730150038/VM

If your backup is OK, then you can also use XSIGR utility to restore a full disk from the backup server.

[root@localhost ~]# xsigr --interactive
--------------------------------------------------------------------------------
|                                                                              |
|                 (c)XSIBackup Granular Restore for Linux                      |
|           (c)2022 | 33HOPS, Sistemas de Información y Redes, SL              |
|                          Daniel J. Garcia Fidalgo                            |
|                            ALL RIGHTS RESERVED                               |
|                                                                              |
--------------------------------------------------------------------------------
System type is: Linux localhost.localdomain 4.18.0-348.12.2.el8_5.x86_64
1 SMP Wed Jan 19 17:53:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
--------------------------------------------------------------------------------
Usage: xsigr --interactive
       xsigr --install
       xsigr --reset
--------------------------------------------------------------------------------
(i) Entering interactive mode
--------------------------------------------------------------------------------
Please enter a path to some (c)XSIBackup repository: [enter to select]: /nas/xsibackup/weekly/
List of restore points:
--------------------------------------------------------------------------------
1: 20230802165000
2: 20230801165000
3: 20230731202730
4: 20230731165000
5: 20230731110144
6: 20230731105249
7: 20230731104919
8: 20230731104720
9: 20230731104408
10: 20230731103601
...
--------------------------------------------------------------------------------
Enter the restore point number to access: 1
--------------------------------------------------------------------------------
Selected restore point is: /nas/xsibackup/weekly/20230802165000
--------------------------------------------------------------------------------
Mounting...
Checking if mount point already exists...
The restore point mount was created at: /nas/xsibackup/weekly/20230802165000
Listing contents:
List of Virtual Machines in the restore point:
--------------------------------------------------------------------------------
1: LAS01
--------------------------------------------------------------------------------
Enter the VM number to browse: 1
--------------------------------------------------------------------------------
Selected Virtual Machine is: LAS01
--------------------------------------------------------------------------------
Choose some -flat.vmdk disk to browse:
List of Virtual Disks:
--------------------------------------------------------------------------------
1: LAS01_0-flat.vmdk
--------------------------------------------------------------------------------

(*) Part of the details above, like the restore points and VM are fake, but will illustrate the process.

In the above example you can access all the VM files at [b]/mnt/XSI/repos/repo01-1MB/20230802165000/LAS01[/b] and restore whatever you need.

[root@localhost ~]# ls -la /mnt/XSI/repos/repo01-1MB/20230802165000/LAS01
total 0
drwxr-xr-x 2 root root          0 Aug  3 05:12 .
drwxr-xr-x 2 root root          0 Aug  3 05:12 ..
-rw-r--r-- 1 root root 8589934592 Aug  3 05:12 LAS01_0-flat.vmdk
-rw-r--r-- 1 root root        586 Aug  3 05:12 LAS01_0.vmdk
-rw-r--r-- 1 root root        131 Aug  3 05:12 LAS01.info
-rw-r--r-- 1 root root       8684 Aug  3 05:12 LAS01.nvram
-rw-r--r-- 1 root root         44 Aug  3 05:12 LAS01.vmsd
-rw-r--r-- 1 root root         44 Aug  3 05:12 LAS01.vmsd.tmp
-rw-r--r-- 1 root root       3024 Aug  3 05:12 LAS01.vmx
-rw-r--r-- 1 root root       3024 Aug  3 05:12 LAS01.vmx.tmp
-rw-r--r-- 1 root root     220171 Aug  3 05:12 vmware-10.log
-rw-r--r-- 1 root root     220904 Aug  3 05:12 vmware-5.log
-rw-r--r-- 1 root root     214890 Aug  3 05:12 vmware-6.log
-rw-r--r-- 1 root root     329338 Aug  3 05:12 vmware-7.log
-rw-r--r-- 1 root root     228158 Aug  3 05:12 vmware-8.log
-rw-r--r-- 1 root root     314320 Aug  3 05:12 vmware-9.log
-rw-r--r-- 1 root root     380905 Aug  3 05:12 vmware.log

#47 (c)XSIBackup-Pro & DC bug tracker » Partially deleting a remote replica dir can result in loss of data » 2023-07-30 15:28:14

admin
Replies: 0

If some remote replica dir is deleted partially, per instance, if you delete some -flat.vmdk file but you don't delete the .map directory containing the data definitions, subsquent jobs running on the same remote dir will result in data loss, as the client will believe the files defined by the .blocklog files inside the .map dir do exist.

Some versions of ©ESXi do just that when you remove some VM from disk. The whole VM folder is not removed as it should happen and an indetermined number of files remain in the dir. Thus when you remove some VM do make sure that the whole VM dir is wiped from the disk.

This will be solved by enriching the replica check algorithm in future versions.

#48 Re: General matters » Restore Backup » 2023-07-20 10:10:56

The general structure of a backup repository is:

-repo-root
-.xsitools
|-data
|-data/.blocklog
|-20230720010101
|-20230720020202
|-20230720030303
|-20230720010101
|-20230720010101/VM1
|-20230720010101/VM2
|-20230720010101/VM3

The source of a restore command must be some VM folder, for example.

xsibackup --restore /backup-volume/repo-root/20230720010101/VM1 /vmfs/volumes/datastore1/restores/VM1

#49 Re: General matters » Issues after upgrade » 2023-07-04 10:09:05

TrueNAS is a closed down environment built on top of FreeBSD, which is not Linux. (c)XSIBackup has been designed to work with Linux OSs. There isn't a server component for FreeBSD and even if there was you would still be limited by a closed down OS designed to work as a user program.

Thanks to being an extremely flexible tool, you could even achieve backing up to USB key attached to a Smart TV over Wi-Fi, if that was your only goal. Nonetheless, in creating exotic configurations you will loose speed, reliability and even some features such as the possibility to do granular restores.

Devices such as TrueNAS are OK for novices or to have at home, they won't be close in possibilities to using a full fledged Linux OS though.

On the other side, adding a disk, formatting it and connecting to a Rocky Linux 8 minimal install is even easier than setting up a device such as TrueNAS.

1/ OpenSSH comes configured by default and with port 22 open in the firewall, so you have to do nothing.
2/ If you install to a big disk you won't even have to configure it, you can directly run --add-key against the Linux backup host IP and start using.
3/ In case you want to add additional disks you can just run a one liner to have it partitioned and formatted as XFS.

#50 Re: General matters » Issues after upgrade » 2023-07-03 19:01:08

There are many things to comment in your post, we'll go from conceptual to more specific:

- You can't backup to iSCSI. We'll reformulate it to: [b]you should not backup to iSCSI[/b].
Why?: because iSCSI volumes are formatted as VMFS. VMFS is a file system designed to host VMs, namely: a few huge virtual disks, not millions of small data chunks, which is what an ©XSIBackup repository is. VMFS is excruciatingly slow when compared to XFS or ext4.

This is unimportant when you use the --replica command, as replicating a VM is just copying the virtual disks somewhere else.

When you use the older Classic versions, the block size is 50MB, but that does not achieve much compression, that's why you achieve a much higher speed, cause in this case, as the block size is 50MB when compared with the 1MB block size of ©XSIBackup-DC, the speed of the FS is not important.

Use NFS with an underlying XFS or ext4 file system, or even better, use some Linux host over IP as the backup host, NFS will be slow when compared to using a full Linux host. The backup host can be a virtual machine, and you can backup that VM, even use CBT on it, which is a good way to achieve geographical dispersion of the data.

To backup over ip to server a.b.c.d on port 22 (SSH):

./xsibackup --backup "VMs(Windows 10 x64 GoodSync)" root@a.b.c.d:22:/home/backup/repositories/repo01

You need to previously link to that server by RSA key:

./xsibackup --add-key root@a.b.c.d:22

- Your .blocklog data isse is due, most probably, to that you started to copy some VM and you then stopped the job. As a result the initial .blocklog file could not be created. The solution is very simple, delete the remote repository and let ©XSIBackup-DC create it again.

If on the contrary it is an old repository, run:

./xsibackup --repair /vmfs/volumes/backup/repo01

The above command will repair the repository and recreate the .blocklog data file. Again, running the --repair command from a Linux backup server would be much faster.

NFS or iSCSI are not local storage. FS commands have to travel over a network. When you are treating with millions of deduplicated blocks the latency can sum up and make things take much longer than having an scheme in which you have an ©ESXi host where you run the client and you copy your backup data to a remote server over IP.

ESXi => Linux Backup host

In such cases things are optimized so that latency does not matter much. The load is balanced between the client and the host, as some ©XSIBackup tasks are done at the host, thus you have two CPUs, plus multiple cores per CPU.

When you have a dedicated backup host you can run the heavy commands directly in the host: --repair, --check, --prune and they will run much faster. You don't need a powerful server as a backup server. A 6th generation Intel i5 is more than enough. The important thing here is that disk latency is in the order of hundreds to thousands of times lower than network latency.

Board footer