#1 2019-08-19 12:20:30

lievenmoors
Member
Registered: 2018-11-21
Posts: 10

xsitools backup failing

Hi,

My Xsitools backups have been working flawlessly for months now.
Suddenly today, it's failing with the errors shown below.
Do you have any idea what is going wrong here?

Thanks for your help,

lieven

----

df -h

VMFS-5       3.9T   3.6T    347.7G  91% /vmfs/volumes/Backup
VMFS-5       1.9T   1.4T    448.2G  76% /vmfs/volumes/datastore1

----

cat jobs/01

"/vmfs/volumes/datastore1/xsi-dir/xsibackup" \
--backup-prog=xsitools:z \
--backup-point=/vmfs/volumes/Backup/xsit-repo \
--backup-type=Custom \
--backup-vms="terra!terra_1.vmdk,aqua,moles" \
--backup-room=2000 \
--mail-to=backups@jes.be \
--use-smtp=1 \
--backup-how=Hot \
--backup-id=01 \
--description="backup all vms" \
--exec=yes >> "/vmfs/volumes/datastore1/xsi-dir/var/logs/xsibackup.log"

----

from var/logs/xsibackup.log

2019-08-19T10:31:40|  [terra] CLXSITO1 error: /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: can't create /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
2019-08-19T10:31:40|  mkdir: can't create directory '/vmfs/volumes/Backup/': Input/output error
2019-08-19T10:31:40|  sed: /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
2019-08-19T10:31:40|  /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: can't create /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
2019-08-19T10:31:40|  mkdir: can't create directory '/vmfs/volumes/Backup/': Input/output error
2019-08-19T10:31:40|  sed: /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
2019-08-19T10:31:40|  /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: can't create /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
2019-08-19T10:31:40|  mkdir: can't create directory '/vmfs/volumes/Backup/': Input/output error
2019-08-19T10:31:40|  sed: /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
2019-08-19T10:31:40|  /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: can't create /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
...And many more of these...

----

In the email sent xsibackup, there is more info:

Last warning for the above VM:
[terra] Info: disks [terra_1.vmdk] have been excluded from the backup
Available space in device /vmfs/volumes/Backup/xsit-repo after backup: 347 Gb.
Complete backup elapsed time: 698 min
The ESXi configuration was saved to /vmfs/volumes/Backup/xsit-repo/20190818230003
• /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: 28090: not found
• /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: 28091: not found
• /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: 28092: not found
• /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: 28093: not found
• /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: 28094: not found
• /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: 28095: not found
• /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: 28096: not found

...And many more of these...

• /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: can't create /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
• /vmfs/volumes/datastore1/xsi-dir/xsibackup: line 155: can't create /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error

• [ Mon Aug 19 10:31:40 UTC 2019 ] ERROR (CLXSITO1), details [terra] error: backup error, details: /vmfs/ volumes/datastore1/xsi-dir/xsibackup: line 155: can't create /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
• [ Sun Aug 18 23:01:05 UTC 2019 ] ERROR (MKROOM01), details Error: cannot make 219G of room, only 178G can be made available

• mkdir: can't create directory '/vmfs/volumes/Backup/': Input/output error
• sed: /vmfs/volumes/Backup/xsit-repo/20190818230003/terra/terra_0-flat.vmdk: Input/output error
• [ Mon Aug 19 02:15:57 UTC 2019 ] WARNING (XSIPRU00), details The (c)XSITools repo was pruned, 87571 Mb were made available. The size of the repo was reduced to 2067832

• [ Mon Aug 19 02:27:55 UTC 2019 ] WARNING (XSIPRU00), details The (c)XSITools repo was pruned, 228348 Mb were made available. The size of the repo was reduced to 1839484

• [ Mon Aug 19 02:39:25 UTC 2019 ] WARNING (VMBDEXC1), details [terra] Info: disks [terra_1.vmdk] have been excluded from the backup

• [ Mon Aug 19 09:20:16 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: /vmfs/volumes/datastore1/xsi-dir

• [ Mon Aug 19 09:20:18 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: /vmfs/volumes/datastore1/xsi-dir

• [ Mon Aug 19 09:20:19 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: /vmfs/volumes/datastore1/xsi-dir

• [ Mon Aug 19 09:20:21 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: /vmfs/volumes/datastore1/xsi-dir

• [ Mon Aug 19 09:20:23 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: /vmfs/ volumes/datastore1/xsi-dir

•  [ Mon Aug 19 09:20:25 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: /vmfs/volumes/datastore1/xsi-dir

• [ Mon Aug 19 09:20:27 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: /vmfs/volumes/datastore1/xsi-dir

• [ Mon Aug 19 09:20:29 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: /vmfs/volumes/datastore1/xsi-dir

• [ Mon Aug 19 09:20:31 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: / vmfs/volumes/datastore1/xsi-dir

• [ Mon Aug 19 09:20:32 UTC 2019 ] WARNING (XSITBPWC), details Cannot process block 9429cca18262d9c1e5f210ce23e30b3c14cec059 (28091), this is just a warning, will try again. Warning: /vmfs/volumes/datastore1/xsi-dir

Offline

#2 2019-08-19 14:25:53

admin
Administrator
Registered: 2017-04-21
Posts: 1,059

Re: xsitools backup failing

That looks like a full or broken disk. This kind of message Input/output error usually means that the storage layer is not able to handle the request for some reason, as said, that usually has to do with full or faulty disks.

If the datastore /vmfs/volumes/Backup is some kind of external device (NFS, iSCSI) it could also mean some cable got disconnected.

Offline

#3 2019-08-26 08:18:58

lievenmoors
Member
Registered: 2018-11-21
Posts: 10

Re: xsitools backup failing

Ok, sorry for the fuzz. Everything went back to normal. Probably a hiccup in the datacenter...
Thanks for your quick support!

Offline

Board footer