Issues after upgrade

marco · 2023-07-03 07:20:42

I wasn't quite sure what title to give this as I seem to have a couple of issues;

Setup;
Source
a) DL360 G9 running VMWare 6.7
b) DL360 G8 running VMWare 7 (I know this isn't 'supported' by VMWare - but it is a stepping stone in my upgrade process)
Destination
DL360 G8 running TrueNAS
TrueNAS hosts an iSCSI volume named backup... source machines connect to that and mount the datastore 'backup' over iSCSI.
XSIBackup-DC 1.7.0.3, licenced
Switch is Cisco, but only 1GB at present.

Question 1:
On host a) I am seeing a transfer speed of only around 4MB/s when using XSIBackup-DC 1.7.0.3
On the same host, if I run XSIBackup-Pro 11.2.19, I am seeing 82MB/s

My command; ./xsibackup --backup "VMs(RUNNING)" /vmfs/volumes/backup/vmhost0 --compression="no"

I am broadly aware of things that can affect speed - but again this is a test on the SAME hardware - one backup scheduled from the command line as a test, the other the 'normal' cron scheduled one.

Question 2;
I was going to run a comparative test on the second server, but I am running in to the .blocking data issue;

Backup folder '/vmfs/volumes/backup/vmhost1/20230702140010'
-------------------------------------------------------------------------------------------------------------
Obtaining .blocklog data from backup repo...
-------------------------------------------------------------------------------------------------------------
| 2023-07-02T14:00:21 | Error code 4524 at file common.c, line 4524 | Error description: |
| could not find .blocklog_dis file, is_first_backupn.c, line 4524 | Error description: |
| (0) |
-------------------------------------------------------------------------------------------------------------
| 2023-07-02T14:00:21 | Error code 3771 at file xsibackup.c, line 3771 | Error description: |
| something went wrong grabbing .blocklog data

I will look at this in more detail once I have a solution to the speed issue.

admin · 2023-07-03 19:01:08

There are many things to comment in your post, we'll go from conceptual to more specific:

- You can't backup to iSCSI. We'll reformulate it to: you should not backup to iSCSI.
Why?: because iSCSI volumes are formatted as VMFS. VMFS is a file system designed to host VMs, namely: a few huge virtual disks, not millions of small data chunks, which is what an ©XSIBackup repository is. VMFS is excruciatingly slow when compared to XFS or ext4.

This is unimportant when you use the --replica command, as replicating a VM is just copying the virtual disks somewhere else.

When you use the older Classic versions, the block size is 50MB, but that does not achieve much compression, that's why you achieve a much higher speed, cause in this case, as the block size is 50MB when compared with the 1MB block size of ©XSIBackup-DC, the speed of the FS is not important.

Use NFS with an underlying XFS or ext4 file system, or even better, use some Linux host over IP as the backup host, NFS will be slow when compared to using a full Linux host. The backup host can be a virtual machine, and you can backup that VM, even use CBT on it, which is a good way to achieve geographical dispersion of the data.

To backup over ip to server a.b.c.d on port 22 (SSH):

./xsibackup --backup "VMs(Windows 10 x64 GoodSync)" root@a.b.c.d:22:/home/backup/repositories/repo01

You need to previously link to that server by RSA key:

./xsibackup --add-key root@a.b.c.d:22

- Your .blocklog data isse is due, most probably, to that you started to copy some VM and you then stopped the job. As a result the initial .blocklog file could not be created. The solution is very simple, delete the remote repository and let ©XSIBackup-DC create it again.

If on the contrary it is an old repository, run:

./xsibackup --repair /vmfs/volumes/backup/repo01

The above command will repair the repository and recreate the .blocklog data file. Again, running the --repair command from a Linux backup server would be much faster.

NFS or iSCSI are not local storage. FS commands have to travel over a network. When you are treating with millions of deduplicated blocks the latency can sum up and make things take much longer than having an scheme in which you have an ©ESXi host where you run the client and you copy your backup data to a remote server over IP.

ESXi => Linux Backup host

In such cases things are optimized so that latency does not matter much. The load is balanced between the client and the host, as some ©XSIBackup tasks are done at the host, thus you have two CPUs, plus multiple cores per CPU.

When you have a dedicated backup host you can run the heavy commands directly in the host: --repair, --check, --prune and they will run much faster. You don't need a powerful server as a backup server. A 6th generation Intel i5 is more than enough. The important thing here is that disk latency is in the order of hundreds to thousands of times lower than network latency.

marco · 2023-07-03 19:34:18

Hi,

Thanks for such a comprehensive reply.

Just one point I did miss from my posting; you say

> - You can't backup to iSCSI. We'll reformulate it to: you should not backup to iSCSI.
> Why?: because iSCSI volumes are formatted as VMFS. VMFS is a file system designed to host VMs, namely: a few huge virtual disks, not > millions of small data chunks, which is what an ©XSIBackup repository is. VMFS is excruciatingly slow when compared to XFS or ext4.
>
> This is unimportant when you use the --replica command, as replicating a VM is just copying the virtual disks somewhere else.

Yes, I did at this stage intend to make a 'replica' of the VMs and was not trying to make deduplicated or incremental backups.

However, I understand what you mean.
With the target machine being TrueNAS (running on bare metal), it should be easy for me to remove my iSCSI volume named 'backup', and create simple naitive storage accessible via ssh.

I shall make these changes and retest the speed... it was just odd that -PRO performs very differently from -DC with basically the same source and target.

Thanks again

admin · 2023-07-04 10:09:05

TrueNAS is a closed down environment built on top of FreeBSD, which is not Linux. (c)XSIBackup has been designed to work with Linux OSs. There isn't a server component for FreeBSD and even if there was you would still be limited by a closed down OS designed to work as a user program.

Thanks to being an extremely flexible tool, you could even achieve backing up to USB key attached to a Smart TV over Wi-Fi, if that was your only goal. Nonetheless, in creating exotic configurations you will loose speed, reliability and even some features such as the possibility to do granular restores.

Devices such as TrueNAS are OK for novices or to have at home, they won't be close in possibilities to using a full fledged Linux OS though.

On the other side, adding a disk, formatting it and connecting to a Rocky Linux 8 minimal install is even easier than setting up a device such as TrueNAS.

1/ OpenSSH comes configured by default and with port 22 open in the firewall, so you have to do nothing.
2/ If you install to a big disk you won't even have to configure it, you can directly run --add-key against the Linux backup host IP and start using.
3/ In case you want to add additional disks you can just run a one liner to have it partitioned and formatted as XFS.

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software

#1 2023-07-03 07:20:42

Issues after upgrade

#2 2023-07-03 19:01:08

Re: Issues after upgrade

#3 2023-07-03 19:34:18

Re: Issues after upgrade

#4 2023-07-04 10:09:05

Re: Issues after upgrade

Board footer