You are not logged in.
Hi,
We are trying to restore a backup over IP. However, we encountered Segmentation fault without any other error logs.
./xsibackup --restore root@backupserver:22:/nas/xsibackup/weekly/20230723150052/VM1/ /vmfs/volumes/datastore1/RESTORE
Tested with the latest binary
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-Free 1.7.0.5: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
Shell binary pool memory size: 2048 MB
-------------------------------------------------------------------------------------------------------------
Shell memory pool size was set to 2048 MB by default
-------------------------------------------------------------------------------------------------------------
Total available memory: 123856 MB (~1GB is always kept as a security margin)
-------------------------------------------------------------------------------------------------------------
Remote system: Linux
-------------------------------------------------------------------------------------------------------------
PID: 4846478, Running job as: root
-------------------------------------------------------------------------------------------------------------
File <xsib64> was updated at root@128.1.255.254:22:/home/root/.xsi/bin
-------------------------------------------------------------------------------------------------------------
File <xsibackup> was updated at root@128.1.255.254:22:/home/root/.xsi/bin
-------------------------------------------------------------------------------------------------------------
File <xsilib> was updated at root@128.1.255.254:22:/home/root/.xsi/bin
-------------------------------------------------------------------------------------------------------------
Restoring VM1 from 128.1.255.254
-------------------------------------------------------------------------------------------------------------
Found remote .xsitools file at: /nas/xsibackup/weekly/.xsitools
-------------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-------------------------------------------------------------------------------------------------------------
Segmentation fault
Tested with our licensed version
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.5.2.1: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-------------------------------------------------------------------------------------------------------------
Shell binary pool memory size: 2048 MB
-------------------------------------------------------------------------------------------------------------
Total available memory: 123863 MB (~1GB is always kept as a security margin)
-------------------------------------------------------------------------------------------------------------
Remote system: Linux
-------------------------------------------------------------------------------------------------------------
PID: 4946927, Running job as: root
-------------------------------------------------------------------------------------------------------------
Restoring VM1 from 128.1.255.254
-------------------------------------------------------------------------------------------------------------
Found remote .xsitools file at: /nas/xsibackup/weekly/.xsitools
-------------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-------------------------------------------------------------------------------------------------------------
Segmentation fault
Please advise
Thanks
Last edited by indieair (2023-08-02 17:24:46)
Offline
You could have run out of TMP space. As your version is a bit outdated we would update or at least try the restore with a more recent version.
To see the details on what is making the restore process to SEGFAULT prepend strace to your restore command.
strace ./xsibackup --restore root@backupserver:22:/nas/xsibackup/weekly/20230723150052/VM1/ /vmfs/volumes/datastore1/RESTORE
To check your backup at the backup server run this command.
xsibackup --check /nas/xsibackup/weekly/20230730150038/VM
If your backup is OK, then you can also use XSIGR utility to restore a full disk from the backup server.
[root@localhost ~]# xsigr --interactive
--------------------------------------------------------------------------------
| |
| (c)XSIBackup Granular Restore for Linux |
| (c)2022 | 33HOPS, Sistemas de Información y Redes, SL |
| Daniel J. Garcia Fidalgo |
| ALL RIGHTS RESERVED |
| |
--------------------------------------------------------------------------------
System type is: Linux localhost.localdomain 4.18.0-348.12.2.el8_5.x86_64
1 SMP Wed Jan 19 17:53:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
--------------------------------------------------------------------------------
Usage: xsigr --interactive
xsigr --install
xsigr --reset
--------------------------------------------------------------------------------
(i) Entering interactive mode
--------------------------------------------------------------------------------
Please enter a path to some (c)XSIBackup repository: [enter to select]: /nas/xsibackup/weekly/
List of restore points:
--------------------------------------------------------------------------------
1: 20230802165000
2: 20230801165000
3: 20230731202730
4: 20230731165000
5: 20230731110144
6: 20230731105249
7: 20230731104919
8: 20230731104720
9: 20230731104408
10: 20230731103601
...
--------------------------------------------------------------------------------
Enter the restore point number to access: 1
--------------------------------------------------------------------------------
Selected restore point is: /nas/xsibackup/weekly/20230802165000
--------------------------------------------------------------------------------
Mounting...
Checking if mount point already exists...
The restore point mount was created at: /nas/xsibackup/weekly/20230802165000
Listing contents:
List of Virtual Machines in the restore point:
--------------------------------------------------------------------------------
1: LAS01
--------------------------------------------------------------------------------
Enter the VM number to browse: 1
--------------------------------------------------------------------------------
Selected Virtual Machine is: LAS01
--------------------------------------------------------------------------------
Choose some -flat.vmdk disk to browse:
List of Virtual Disks:
--------------------------------------------------------------------------------
1: LAS01_0-flat.vmdk
--------------------------------------------------------------------------------
(*) Part of the details above, like the restore points and VM are fake, but will illustrate the process.
In the above example you can access all the VM files at [b]/mnt/XSI/repos/repo01-1MB/20230802165000/LAS01[/b] and restore whatever you need.
[root@localhost ~]# ls -la /mnt/XSI/repos/repo01-1MB/20230802165000/LAS01
total 0
drwxr-xr-x 2 root root 0 Aug 3 05:12 .
drwxr-xr-x 2 root root 0 Aug 3 05:12 ..
-rw-r--r-- 1 root root 8589934592 Aug 3 05:12 LAS01_0-flat.vmdk
-rw-r--r-- 1 root root 586 Aug 3 05:12 LAS01_0.vmdk
-rw-r--r-- 1 root root 131 Aug 3 05:12 LAS01.info
-rw-r--r-- 1 root root 8684 Aug 3 05:12 LAS01.nvram
-rw-r--r-- 1 root root 44 Aug 3 05:12 LAS01.vmsd
-rw-r--r-- 1 root root 44 Aug 3 05:12 LAS01.vmsd.tmp
-rw-r--r-- 1 root root 3024 Aug 3 05:12 LAS01.vmx
-rw-r--r-- 1 root root 3024 Aug 3 05:12 LAS01.vmx.tmp
-rw-r--r-- 1 root root 220171 Aug 3 05:12 vmware-10.log
-rw-r--r-- 1 root root 220904 Aug 3 05:12 vmware-5.log
-rw-r--r-- 1 root root 214890 Aug 3 05:12 vmware-6.log
-rw-r--r-- 1 root root 329338 Aug 3 05:12 vmware-7.log
-rw-r--r-- 1 root root 228158 Aug 3 05:12 vmware-8.log
-rw-r--r-- 1 root root 314320 Aug 3 05:12 vmware-9.log
-rw-r--r-- 1 root root 380905 Aug 3 05:12 vmware.log
Offline
Thanks for the response.
I tried to check and realise there are many missing blocks. Does it means that the backup was corrupted to begin with?
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-Free 1.7.0.5: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: Linux, Kernel 3 Major 10 Minor 0 Patch 1160
-------------------------------------------------------------------------------------------------------------
PID: 709, Running job as: root
-------------------------------------------------------------------------------------------------------------
Repository Check: 16 files
-------------------------------------------------------------------------------------------------------------
Check type is file (only block existence is checked)
-------------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS GENERAL MISSED
-------------------------------------------------------------------------------------------------------------
1/16 k8-022.vmx.map 3.09 KB | 100.00% 0.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:49 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
2/16 k8-022-flat.vmdk.map 30.00 GB | 100.00% 99.99% 23562 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
3/16 k8-022.vmdk.map 526.00 B | 100.00% 99.99% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
4/16 k8-022.vmsd.map 44.00 B | 100.00% 99.99% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
5/16 vmware-2.log.map 247.06 KB | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
6/16 k8-022.nvram.map 8.48 KB | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
7/16 k8-022-f30c9564.hlog.map 53.00 B | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
8/16 vmware-1.log.map 263.34 KB | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
9/16 k8-022.vmxf.map 47.00 B | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
10/16 vmware-4.log.map 235.80 KB | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
11/16 vmware-3.log.map 303.02 KB | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
12/16 k8-022.vmx~.map 3.09 KB | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
13/16 vmware.log.map 718.80 KB | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
14/16 k8-022.vmsd.tmp.map 44.00 B | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
15/16 k8-022.vmx.tmp.map 3.09 KB | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
16/16 k8-022-Snapshot27.vmsn.map 27.80 KB | 100.00% 100.00% 1 KO
-------------------------------------------------------------------------------------------------------------
| 2023-08-03T18:32:50 | Error code 3619 at file xsibackup.c, line 3619 | Error description: |
| there are missing blocks |
-------------------------------------------------------------------------------------------------------------
Removed PID OK
-------------------------------------------------------------------------------------------------------------
Offline
I tried to restore locally but also encountered can't find blocks in .map file.
Is there anyway to recover the data file? It is happening to all the current backups too
-------------------------------------------------------------------------------------------------------------
::: detail ::: 0.75% done | block 612 out of 81920 | Done 0.75%
-------------------------------------------------------------------------------------------------------------
| 2023-08-14T11:23:20 | Error code 786 at file restore.c, line 786 | Error description: |
| can't find block d4c8763c5674fcc53c3e75e23d86ac300842befe at pos 612 in .map file /nas/xsibackup/ |
| /weekly/20230813150012/VM1/VM1.vmdk.map |
-------------------------------------------------------------------------------------------------------------
::: detail ::: 0.75% done | block 613 out of 81920 | Done 0.75%
-------------------------------------------------------------------------------------------------------------
| 2023-08-14T11:23:20 | Error code 786 at file restore.c, line 786 | Error description: |
| can't find block db8758149ed7596e747da384e2b785a64105378f at pos 613 in .map file /nas/xsibackup/ |
| /weekly/20230813150012/VM1/VM1.vmdk.map |
-------------------------------------------------------------------------------------------------------------
::: detail ::: 0.75% done | block 614 out of 81920 | Done 0.75%
-------------------------------------------------------------------------------------------------------------
Offline
If you have missing blocks you can't create them out of thin air. You should check your repos from time to time to prevent this kind of thing from happening. We can't assess what is the reason for the block loss, it could be from a damaged disk to incomplete backups that returned some errors or a partially deleted repository.
First of all you should use some newer version to restore. Restore functionality is not limited by any license check, thus you can use the free version to restore.
If you use the --force argument at the end of your restore command you will be able to restore your disk without the missing blocks, just holes instead of those blocks.
If you keep getting a segfault, then the repo could be broken. The reason for that could be missing or damaged files or them having been edited with some text editor that changed the line ends or the file encoding.
You should check that you have an .xsitools file in the root of the repo that you can read with cat or vi, a data folder with a block manifest named .blocklog and the backup folders in the repo root with .map files that contain the constituent blocks.
.xsitools file
Desc: XSITools Repo v 2.0.0
Bsiz: 1048576
Comp: 1
The data folder contains a hierachical structure with the block data in a six level nested folder set.
[root@localhost ~]# ls -la /home/backup/repositories/repo01/data/
total 360
drwxr-xr-x. 18 root root 167 Aug 15 22:51 .
drwxr-xr-x. 8 root root 145 Aug 15 22:51 ..
drwxr-x---. 18 root root 150 Aug 15 18:51 0
drwxr-x---. 18 root root 150 Aug 15 18:52 1
drwxr-x---. 18 root root 150 Aug 15 18:51 2
drwxr-x---. 18 root root 150 Aug 15 18:51 3
drwxr-x---. 18 root root 150 Aug 15 18:51 4
drwxr-x---. 18 root root 150 Aug 15 18:51 5
drwxr-x---. 18 root root 150 Aug 15 18:51 6
drwxr-x---. 18 root root 150 Aug 15 18:51 7
drwxr-x---. 18 root root 150 Aug 15 18:51 8
drwxr-x---. 18 root root 150 Aug 15 18:52 9
drwxr-x---. 18 root root 150 Aug 15 18:52 a
drwxr-x---. 18 root root 150 Aug 15 18:52 b
-rw-r--r--. 1 root root 366121 Aug 15 22:51 .blocklog
drwxr-x---. 18 root root 150 Aug 15 18:52 c
drwxr-x---. 18 root root 150 Aug 15 18:51 d
drwxr-x---. 18 root root 150 Aug 15 18:51 e
drwxr-x---. 18 root root 150 Aug 15 18:52 f
Some repo restore point folder:
[root@localhost ~]# ls -la /home/backup/repositories/repo01/20230815227220/WXPMK
total 1048
drwxr-xr-x. 2 root root 4096 Aug 15 22:51 .
drwxr-xr-x. 3 root root 19 Aug 15 22:51 ..
-rw-r--r--. 1 root root 97 Aug 15 22:51 vmware-10.log.map
-rw-r--r--. 1 root root 97 Aug 15 22:51 vmware-11.log.map
-rw-r--r--. 1 root root 146 Aug 15 22:51 vmware-12.log.map
-rw-r--r--. 1 root root 48 Aug 15 22:51 vmware-13.log.map
-rw-r--r--. 1 root root 97 Aug 15 22:51 vmware-14.log.map
-rw-r--r--. 1 root root 48 Aug 15 22:51 vmware-15.log.map
-rw-r--r--. 1 root root 48 Aug 15 22:51 vmware.log.map
-rw-r--r--. 1 root root 1003520 Aug 15 22:51 WXPMK-flat.vmdk.map
-rw-r--r--. 1 root root 118 Aug 15 22:51 WXPMK.info
-rw-r--r--. 1 root root 45 Aug 15 22:51 WXPMK.info.map
-rw-r--r--. 1 root root 46 Aug 15 22:51 WXPMK.nvram.map
-rw-r--r--. 1 root root 45 Aug 15 22:51 WXPMK.vmdk.map
-rw-r--r--. 1 root root 44 Aug 15 22:51 WXPMK.vmsd.map
-rw-r--r--. 1 root root 44 Aug 15 22:51 WXPMK.vmsd.tmp.map
-rw-r--r--. 1 root root 46 Aug 15 22:51 WXPMK.vmx.map
-rw-r--r--. 1 root root 46 Aug 15 22:51 WXPMK.vmx~.map
-rw-r--r--. 1 root root 46 Aug 15 22:51 WXPMK.vmx.tmp.map
Detail of some .map file
[root@localhost ~]# tail -n20 /home/backup/repositories/repo01/20230815227220/WXPMK/WXPMK-flat.vmdk.map
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
6bf4834b3c2b6d46e03886a99b31e75b5dc6df00;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3;1048576
The end of this -flat.vmdk file contains mostly 1MB zeroed blocks (3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3), thence the hash repetition.
Offline
Hi,
Thanks for the information.
We have found out that when we use the --rotate option in our backup, the blocks will be missing after the first rotation
Our command we used is
/scratch/XSI/XSIBackup-DC/xsibackup --backup "VMs(ALL)" user@remoteserver:22:/nas/xsibackup/all --rotate=4
After the 5th run which will perform a rotation by removing the oldest backup, the blocks will be missing.
Is there anyway we can rotate the backups safely?
Offline
We have repos in clients that have been rotating for years and all non pruned blocks are there.
The most obvious explanation is that you may be trying to restore some restore point that was indeed pruned but for some reason the .map files weren't deleted at the end of the prune process.
Please contact our support department with as much details as possible, we will check the exact same kind of job in our lab in the meanwhile.
Offline
We have been running a backup job similar to yours for some hours now.
n=0;while [ "$n" -lt "100" ];do xsibackup --backup "VMs(ALL)" "root@a.b.c.d:22:/home/backup/repo02-1MB" --rotate="4";n=$(( n+1 ));done
We have also been banging the two guests in the ©ESXi lab host with big downloads and then removal of the downloaded files to make sure that the number of blocks would be changing all the time. No quiescing has been performed, thus some files have probably been backed up partially in some of the backup rounds.
We haven't been able to reproduce your issue so far. As commented before we maintain several backup schemes at clients with prunes and --rotate commands that, as you probably already know, is nothing but the pruning of the eldest restore points.
We can offer you some ideas on what might be going on:
1/ Broken disk? just thinking loud.
2/ Broken FS.
3/ Accidental deletion of blocks.
4/ Attempt to restore from some leftover restore point corresponding to some pruned dir.
5/ Broken restore points due to backup tests or failed backups. This would not affect other restore points corresponding to successful backups though.
If you can't find the cause contact support to provide more details.
Offline