You are not logged in.
running 11.2.19 and tried to remove cron from within the gui and it just showed an error. ended up creating a file in the cron folder with the error as its name.
┌─────────────────────────────────────────────────────┐
│ Error removing jobs from /var/spool/cron/crontabs/ │
│ Error: Expected at least 5 tokens for --menu, have │
│ 4. │
│ Use --help to list options. │
│ │
│ │
│ │
│ │
│ │
├─────────────────────────────────────────────────────┤
│ < OK > │
└─────────────────────────────────────────────────────┘
[root@esxi2:/var/spool/cron/crontabs] ls -al
total 12
---------- 1 root root 0 Sep 15 13:28 ?Error: Expected at least 5 tokens for --menu, have 4.?Use --help to list options.
drwxr-xr-x 1 root root 512 Sep 15 13:28 .
drwxr-xr-x 1 root root 512 Sep 15 11:00 ..
-r--r--r-T 1 root root 324 Apr 3 2018 root
it is honestly rather underpowered. 1 GB ram and 2 cpu. plenty of disk. the prune over NFS/remote is working however.
the repo is in the neighborhood of 12 TB with 10 MB block sizes. nightly backups seems to have quite a large delta even though not much data is being changed. not sure what the best alternative would be. i know we could de separate repos and rotate those but i would need much more space that i have right now.
thank you - i understand now.
for now i have just commented out the cron line under /path/to/xsibackup/var/spool/cron/root-crontab and --update-cron. this clears it out successfully.
seems this flag does not work. update and install is fine - it will not remove on any of my hosts.
[root@pro1:/vmfs/volumes/5d75000d-393caa77-6cf5-a0369f3ea540/XSI/pro1/XSIBackup-DC] ./xsibackup --update-cron
-----------------------------------------------------------------------------------------------------------
UPDATE CRON (root)
-----------------------------------------------------------------------------------------------------------
The (c)ESXi cron was updated with (c)XSIBackup-DC cron of user root
-----------------------------------------------------------------------------------------------------------
The ESXi crond service was restarted
-----------------------------------------------------------------------------------------------------------
Removed <tmp> dir OK
-----------------------------------------------------------------------------------------------------------
Removed PID OK
-----------------------------------------------------------------------------------------------------------
[root@pro1:/vmfs/volumes/5d75000d-393caa77-6cf5-a0369f3ea540/XSI/pro1/XSIBackup-DC] ./xsibackup --uninstall-cron
Could not find any cron install for user root
-----------------------------------------------------------------------------------------------------------
The ESXi crond service was restarted
-----------------------------------------------------------------------------------------------------------
Removed <tmp> dir OK
-----------------------------------------------------------------------------------------------------------
Removed PID OK
-----------------------------------------------------------------------------------------------------------
[root@pro1:/vmfs/volumes/5d75000d-393caa77-6cf5-a0369f3ea540/XSI/pro1/XSIBackup-DC] cat /var/spool/cron/crontabs/root
#min hour day mon dow command
1 1 * * * /sbin/tmpwatch.py
1 * * * * /sbin/auto-backup.sh
0 * * * * /usr/lib/vmware/vmksummary/log-heartbeat.py
*/5 * * * * /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
00 1 * * * localcli storage core device purge
0 0 1-31/2 * * "/vmfs/volumes/QNAP3/XSI/pro1/XSIBackup-DC/etc/jobs/001" > /dev/null 2>&1 # Added by XSIBackup
I am running prune on my remote system and have run into an issue where it gets stuck.
it has been running for ~ 8 hours or so. when checking processes i can see sort is still running:
22037 admin 4 S ./xsibackup --prune 20200721000005
23778 admin 2256 S sh -c sort -T'/share/CACHEDEV1_DATA/tmp/xsi/tmp' '/share/CACHEDEV1_DATA/tmp/xsi/tmp/0U.xsi' | uniq > '/share/CACHEDEV1_DATA/tmp/xsi/tmp/0U.xsi.tmp' && mv '/share/CACHEDEV1_DATA/tmp/xsi/tmp/0U.xsi.tmp' '/share/CACHEDEV1_DATA/tmp/xsi/tmp/0U.xsi' && rm -rf '/share/CACHEDEV1_DATA/tmp/xsi/tmp/0U.xsi.tmp' && echo 1 || echo 0
23779 admin 346336 D sort -T/share/CACHEDEV1_DATA/tmp/xsi/tmp /share/CACHEDEV1_DATA/tmp/xsi/tmp/0U.xsi
output of prune is as follows:
[/share/CACHEDEV1_DATA/BKUP] # ./xsibackup --prune 20200721000005
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: Linux, Kernel 4 Major 14 Minor 24 Patch 0
-----------------------------------------------------------------------------------------------------------
License: unlicensed trial version
-----------------------------------------------------------------------------------------------------------
PID: 22037, Running job as: root
-----------------------------------------------------------------------------------------------------------
Finding blocks to prune, please wait...
-----------------------------------------------------------------------------------------------------------
Getting map files from repo...
-----------------------------------------------------------------------------------------------------------
(!) Not enough space to prune in the tmp folder: /tmp. required: 857074968, available: 36831232
-----------------------------------------------------------------------------------------------------------
Looking for alternative locations...
-----------------------------------------------------------------------------------------------------------
The TMP dir was moved to </share/CACHEDEV1_DATA/tmp/xsi/tmp> (515.03 GB free)
-----------------------------------------------------------------------------------------------------------
Retrieving general block data 100.00%
-----------------------------------------------------------------------------------------------------------
Please wait while we order the data at: /share/CACHEDEV1_DATA/tmp/xsi/tmp/0U.xsi
-----------------------------------------------------------------------------------------------------------
top shows this for the sort process:
Mem: 840032K used, 54988K free, 24620K shrd, 39684K buff, 58112K cached
CPU: 3.5% usr 9.4% sys 0.0% nic 7.8% idle 78.3% io 0.0% irq 0.8% sirq
Load average: 5.94 6.04 6.35 2/729 4277
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
23779 23778 admin D 808m 92.2 0 0.2 sort -T/share/CACHEDEV1_DATA/tmp/xsi/tmp /share/CACHEDEV1_DATA/tmp/xsi/tmp/0U.xsi
maybe i should try pruning from a different machine via NFS connection? pruning via ssh/remote does not work from my understanding. issue is that i do not believe i have a server other than this one at the remote site.
any thoughts? thanks.
this is using the --prune feature. but yes understood on your warning
Thought I would share a simple oneliner I have used to purge old backups from the repo. This just uses unix find and modification date. example shows a purge of 7 day old or more backups.
find /path/to/repo -type d -maxdepth 1 -mtime +7 -print0 | xargs -0 -i /path/to/xsibackup --prune {}
have you tested a restore? thats the only way to test a backup.
curious if you have thought about reporting the backup run data via means other than email. For example, this could submit JSON data via a webhook call to a centralized reporting structure - or submit webhook to slack etc. This would be helpful in the case of clients having data collection, reporting and alerting in place and would not rely on manual email checks.
thanks!
It would be useful if the email report for a --backup job could report how much new data was copied versus matched/same.
Also maybe list the start time, as it already shows end time and duration.
Thank you!
Thank you - makes sense. I will check more into this storage array as I still believe that I should be getting faster speed from it. Your explanation makes sense however and I would have to account for this if i was trying to backup a larger VM. The zero block skipping has been working great as this disk is much larger than 4.6 TB.
Thanks.
Now that I have been running things for awhile via cron with reporting I wanted to clarify my expectations towards what I am seeing. All VMs are now backing up via remote/ssh which from your great description above is the most efficient as it skips blocks in the most efficient way. Our VMs backup with a range of speed (all from the same datastore) ranging from 80 MB/sec -> 1000 MB/sec. Our largest VM which currently has ~ 4.6TB of used space is consistently taking 8+ hours to compare even when not much data is changing. I am really curious as to what your expectation on speed in an average scenario would be for this VM and if there is anything else we may do to improve it.
In this case I am going to a remote linux box via ssh so there is not vmkfstools, but i would imagine if i were going to a remote esxi host and running Free or Pro with onediff that it would be complete much faster. I understand that I am comparing quite different backup/replica scenarios though in this example.
Without getting lost in the weeds, do you think that 8+ hours is what i should expect for a VM of this size? What would you do in this scenario if you required a time efficient backup of such vm?
Thank you for your work.
As i understand it, the --rotate argument acts on an entire backup repo irrespective of what VMs have been chosen to be backed up correct?
For example, if i have 2 hosts backing up to the same repo (not using --subfolder) and the first host uses the --rotate argument while the second does not, i should expect the old backups from the second host to be purged when the first backup command rotates correct? Just trying to get an understanding as I may choose to backup some VMs at a less frequent interval due to size etc.
Is there any thought towards perhaps defining the rotation to be respective to the VM level (ie. 7 revisions of each VM, but VM1 backs up weekly, while VM 2 backs up nightly, etc.)
Thank you.
I have modified the given script to account for VMs that may have a space in the name:
OLDIFS="$IFS"
IFS=$'\n'
VMList=$(vim-cmd vmsvc/getallvms | sed '1d' | awk -F' {2,}' '{if ($1 > 0) print $1" "$2}')
for VM in $VMList; do
VMName=$(echo $VM | awk '{$1=""; print $0}' | awk '{$1=$1};1')
VMID=$(echo $VM | awk '{print $1}')
VMMatch=false
for pattern in $@; do
if (echo $VMName | grep -q $pattern)
then VMMatch=true; break
fi
done
if $VMMatch; then continue; fi
if (vim-cmd vmsvc/power.getstate $VMID | grep -q "Powered on")
then VMsToBackup="$VMsToBackup,$VMName"
fi
done
IFS=$OLDIFS
echo ${VMsToBackup#?}
i see. the odd thing is that i was able to make a snapshot via vcenter when xsibackup failed to. it is on iSCSI vmfs6. it seems to be working now once it re-registered.
It is breaking on vmdk when it does. File paths are fine. the above example file is going from:
/vmfs/volumes/5b6105f0-96792252-f5ce-ac1f6b1dc5be/Graylog/Graylog.vmdk
to
root@192.168.246.77:22:/media/wesley
I would also not imagine network issue in this environment as there has been no indicated issue of it previously. Either way what would be the best way to get more data when this failure occurs to help diagnose?
Thanks.
well whatever it was has been cleared up by unregister/re-register. thx.
yeah it is odd - i figure i will go through shutting it down - unregistering and re-registering when i get a moment.
it is in a cluster but all other vms are as well. on VMFS6 and Win 10. I just wish i could see an actual error
command and output for ref
[root@supernaut:~] /vmfs/volumes/SSD2-Super/XSIBackup-DC/xsibackup --backup "VMs(Graylog)" root@192.168.246.77:22:/media/wesley --ssh-ciphers=aes128-c
tr --rotate=3 --verbosity=10
-----------------------------------------------------------------------------------------------------------
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.3.0.2: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-----------------------------------------------------------------------------------------------------------
License: 000506630000000000000000ac1f6b1dc42e
-----------------------------------------------------------------------------------------------------------
Primary TMP folder: /tmp/xsi/2215205
-----------------------------------------------------------------------------------------------------------
Secondary TMP folder: /vmfs/volumes/SSD2-Super/XSIBackup-DC/tmp/2215205
-----------------------------------------------------------------------------------------------------------
SSH Out: tue jun 30 11:08:27 utc 2020
SSH: Tue Jun 30 11:08:28 UTC 2020
-----------------------------------------------------------------------------------------------------------
Remote system: linux
-----------------------------------------------------------------------------------------------------------
User defined ciphers: aes128-ctr
-----------------------------------------------------------------------------------------------------------
PID: 2215205, Running job as: root
-----------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-----------------------------------------------------------------------------------------------------------
Block size is 1048576 bytes (1024.00 KB)
-----------------------------------------------------------------------------------------------------------
Configured ciphers: aes128-ctr
-----------------------------------------------------------------------------------------------------------
Checking whether XSIBackup-DC can be found at </usr/bin/xsibackup>
-----------------------------------------------------------------------------------------------------------
XSIBackup-DC was found at </usr/bin/xsibackup>
-----------------------------------------------------------------------------------------------------------
Created remote folder at 192.168.246.77:22:/media/wesley/20200630110824
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC setting repository at /media/wesley
-----------------------------------------------------------------------------------------------------------
Found remote dir /media/wesley/data at 192.168.246.77:22
-----------------------------------------------------------------------------------------------------------
.xsitools file located at root@192.168.246.77:22:/media/wesley/.xsitools
Ordering blocks 100.00%
-----------------------------------------------------------------------------------------------------------
Hardware disk: /dev/disks/naa.600140562d158eedf443d41afd97afd8
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: Graylog
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : Graylog (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
Created remote folder at 192.168.246.77:22:/media/wesley/20200630110824/Graylog
-----------------------------------------------------------------------------------------------------------
Lock file: /media/wesley/20200630110824/Graylog/.locked
New backup/replica: Graylog | folder
-----------------------------------------------------------------------------------------------------------
Backup start date: 2020-06-30 11:08:42
....
Block 8371, exists: 0, sha1str: 328dbbc21effff1ee2feb95d264adf760a0ecf1c
::: detail ::: 16.35% done | block 8372 out of 51200 | Done 15.06%
Block 8372, exists: 0, sha1str: 3b11fb6c860c0dcfc1c4b54a01b11c9f61a68643
::: detail ::: 16.35% done | block 8373 out of 51200 | Done 15.06%
Block 8373, exists: 0, sha1str: 25b54727cb314bb77d4db9442fbd71b97c919553
::: detail ::: 16.36% done | block 8374 out of 51200 | Done 15.06%
Block 8374, exists: 0, sha1str: 8125e635276bcf3bc169dc20e7db96cb75e5a1bc
::: detail ::: 16.36% done | block 8375 out of 51200 | Done 15.06%
Block 8375, exists: 0, sha1str: bbf583a5cbd60ffd83db201fb2802339ce87de43
-----------------------------------------------------------------------------------------------------------
SIGTERM (13) condition was trapped: check logs for more details
-----------------------------------------------------------------------------------------------------------
Cleaning up...
-----------------------------------------------------------------------------------------------------------
*** Snapshot was removed ***
-----------------------------------------------------------------------------------------------------------
Removed <tmp> dir OK
-----------------------------------------------------------------------------------------------------------
Unlocked backup OK
-----------------------------------------------------------------------------------------------------------
Removed PID OK
-----------------------------------------------------------------------------------------------------------
Intermittently I get SIGTERM (13) condition during a backup. This has been encountered across different hosts, in this case I am using 13.0.2 from ESXi 6.7 to Ubuntu 18 via ssh. If I re-run the backup task immediately it typically goes past the error. I have seen it a handful of times.
I have found the corresponding errors from the receiver end's error.log:
2020-06-26T02:12:56 | Error code 560 at file xsibackup.c, line 560
Error description: can't evaluate ?p?ュg as a local path
2020-06-26T11:24:54 | Error code 560 at file xsibackup.c, line 560
Error description: can't evaluate ?"? as a local path
Error description: can't evaluate )T
_no as a local path
2020-06-30T11:10:15 | Error code 560 at file xsibackup.c, line 560
Error description: can't evaluate ? as a local path
Any ideas? Thanks.
So far I have not found anything to explain it - still looking. There are no .lck files and vmkfstools -D doesnt show a lock on the vmdk file. Either way I can create and remove snapshots manually on the VM. I do not see any errors in logs.
How would you suggest I troubleshoot this issue. I have successfully set up xsibackup DC on 3 hosts and backups are going well. On a single VM, xsibackup fails to create a snapshot and thus does not back up the vmdk files as they are excluded/in-use. The verbose output does not report an error and if i manually create a snapshot prior to backup, the backup runs because a previous snapshot exists.
output below is truncated
[root@localhost:/vmfs/volumes/5d75000d-393caa77-6cf5-a0369f3ea540/XSI] /vmfs/volumes/QNAP3/XSI/pro1/XSIBackup-DC/xsibackup --backup "VMs(Global Viewer I)" admin@172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/ --ssh-ciphers=aes128-gcm@openssh.com --remote-path=/share/CACHEDEV1_DATA/BKUP/xsibackup --verbosity=10
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-----------------------------------------------------------------------------------------------------------
License: XXXX
-----------------------------------------------------------------------------------------------------------
Primary TMP folder: /tmp/xsi/2307404
-----------------------------------------------------------------------------------------------------------
Secondary TMP folder: /vmfs/volumes/QNAP3/XSI/pro1/XSIBackup-DC/tmp/2307404
-----------------------------------------------------------------------------------------------------------
SSH: Sun Jun 28 08:57:11 EDT 2020
-----------------------------------------------------------------------------------------------------------
Remote system: linux
-----------------------------------------------------------------------------------------------------------
User defined ciphers: aes128-gcm@openssh.com
-----------------------------------------------------------------------------------------------------------
PID: 2307404, Running job as: admin
-----------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-----------------------------------------------------------------------------------------------------------
Block size is 10485760 bytes (10.00 MB)
-----------------------------------------------------------------------------------------------------------
Configured ciphers: aes128-gcm@openssh.com
-----------------------------------------------------------------------------------------------------------
Checking whether XSIBackup-DC can be found at </share/CACHEDEV1_DATA/BKUP/xsibackup>
-----------------------------------------------------------------------------------------------------------
XSIBackup-DC was found at </share/CACHEDEV1_DATA/BKUP/xsibackup>
-----------------------------------------------------------------------------------------------------------
Created remote folder at 172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/20200628125711
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC setting repository at /share/CACHEDEV1_DATA/BKUP/
-----------------------------------------------------------------------------------------------------------
Found remote dir /share/CACHEDEV1_DATA/BKUP//data at 172.20.30.40:22
-----------------------------------------------------------------------------------------------------------
.xsitools file located at admin@172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/.xsitools
Ordering blocks 100.00%
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: Global Viewer I
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : Global Viewer I (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
Created remote folder at 172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/20200628125711/Global Viewer I
-----------------------------------------------------------------------------------------------------------
Lock file: /share/CACHEDEV1_DATA/BKUP/20200628125711/Global Viewer I/.locked
New backup/replica: Global Viewer I | folder
-----------------------------------------------------------------------------------------------------------
Backup start date: 2020-06-28 12:57:23
-----------------------------------------------------------------------------------------------------------
2020-06-28 12:57:23 | Backing up 22 files, total size is 545567757997
-----------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-----------------------------------------------------------------------------------------------------------
1/22 Global Viewer I-0459a943.hlog [skipped excluded]
-----------------------------------------------------------------------------------------------------------
2/22 Global Viewer I-flat.vmdk [skipped excluded]
-----------------------------------------------------------------------------------------------------------
3/22 Global Viewer I.vmdk [skipped excluded]
-----------------------------------------------------------------------------------------------------------
4/22 Global Viewer I.nvram [skipped excluded]
....
with manual snapshot:
[root@localhost:/vmfs/volumes/5d75000d-393caa77-6cf5-a0369f3ea540/XSI] /vmfs/volumes/QNAP3/XSI/pro1/XSIBackup-DC/xsibackup --backup "VMs(Global Viewer I)" admin@172.20.30.40:22:/share/CACHEDEV1_DATA/BKUP/ --ssh-ciphers=aes128-gcm@openssh.com --remote-path=/share/CACHEDEV1_DATA/BKUP/xsibackup
|---------------------------------------------------------------------------------|
||-------------------------------------------------------------------------------||
||| (c)XSIBackup-DC 1.3.0.1: Backup & Replication Software |||
||| (c)33HOPS, Sistemas de Informacion y Redes, S.L. | All Rights Reserved |||
||-------------------------------------------------------------------------------||
|---------------------------------------------------------------------------------|
(c)Daniel J. Garcia Fidalgo | info@33hops.com
|---------------------------------------------------------------------------------|
System Information: ESXi, Kernel 6 Major 7 Minor 0 Patch 0
-----------------------------------------------------------------------------------------------------------
License: XXXX
-----------------------------------------------------------------------------------------------------------
SSH: Sun Jun 28 08:56:00 EDT 2020
-----------------------------------------------------------------------------------------------------------
Remote system: linux
-----------------------------------------------------------------------------------------------------------
User defined ciphers: aes128-gcm@openssh.com
-----------------------------------------------------------------------------------------------------------
PID: 2307266, Running job as: admin
-----------------------------------------------------------------------------------------------------------
LZJB compression has been enabled
-----------------------------------------------------------------------------------------------------------
Block size is 10485760 bytes (10.00 MB)
-----------------------------------------------------------------------------------------------------------
(c)XSIBackup-DC setting repository at /share/CACHEDEV1_DATA/BKUP/
-----------------------------------------------------------------------------------------------------------
Virtual Machine Name: Global Viewer I
-----------------------------------------------------------------------------------------------------------
Creating snapshot VM : Global Viewer I (powered on)
-----------------------------------------------------------------------------------------------------------
*** Snapshot was successfully created ***
-----------------------------------------------------------------------------------------------------------
New backup/replica: Global Viewer I | folder
-----------------------------------------------------------------------------------------------------------
Backup start date: 2020-06-28 12:56:12
-----------------------------------------------------------------------------------------------------------
2020-06-28 12:56:12 | Backing up 26 files, total size is 556295208637
-----------------------------------------------------------------------------------------------------------
NUMBER FILE SIZE PROGRESS
-----------------------------------------------------------------------------------------------------------
1/26 Global Viewer I-0459a943.hlog 92.00 B | Done 0.00%
-----------------------------------------------------------------------------------------------------------
2/26 Global Viewer I-flat.vmdk 500.00 GB | Done 0.00%
...
thanks - i understand that option but it will not help my specific use case. i will work around the lack of VM exclusion option another way.
thanks - i would propose you change this message then and exclude remote:
Error description: when the action is --prune, third argument must be a local or remote path string, passed: