Forum ©XSIBackup: ©VMWare ©ESXi Backup Software
MenuYou are not logged in.
- Topics: Active | Unanswered
#451 Re: General matters » using DC to maintain a set of replicas » 2021-05-30 10:58:47
Yes, that's a recommended setup. Please note that you can reduce it to a single job by using some job syntax like the example below:
/scratch/XSI/XSIBackup-DC/xsibackup \
--replica=cbt \
"VMs(RUNNING)" \
/vmfs/volumes/backup/replicas/$(( $(date +%j) % 5 )) \
--options=R \
--use-smtp="1" \
--mail-to=mail-from \
>> /scratch/XSI/XSIBackup-DC/var/log/xsibackup.log 2>&1In the above job the code snippet [b]$(( $(date +%j) % 5 ))[/b] is generating the day of the year 1-365 and getting it's 5 modulus, which would generate
/vmfs/volumes/backup/replicas/0
/vmfs/volumes/backup/replicas/1
/vmfs/volumes/backup/replicas/2
/vmfs/volumes/backup/replicas/3
/vmfs/volumes/backup/replicas/4Five different replicas. You just have to change the modulus operator to have any number of replicas with a single job.
The [b]--options=R[/b] argument keeps a test snapshot on the remote end. Please do understand that this snapshot is used just to store the information generated during a replicated VM test and that it is obliterated on the subsequent CBT cycle. Namely this snapshot is totally removed, none of its information will survive and the VM will always be returned to its replicated state, with an exact mirror copy of the original VM.
If you manipulate this snapshot on your own, like deleting it through the (c)ESXi web interface, instead of letting the --replication process handle it, you will most likely ruin the replication process and you may damage your VM.
If you set some kind of active/passive cluster, you will have to STONITH (Shoot The Other Node In The Head) before turning some replica into the production server.
In automated cluster environments this is done by the cluster management layer. In case of using (c)XSIBackup to keep a set of replicas, you will normally do this by hand, as (c)XSIBackup is more on the side of a backup utility than on that of a real time replication system. Since distributed FSs such as Gluster or Ceph have reached a mature stage, you will normally use this kind of tool as the base of a multinode cluster.
#452 Re: General matters » Error description: can't open output file: /vmfs/volumes/... » 2021-05-28 16:41:13
Forget our previous statement, we messed up. This can then only have to do with your primary temp dir /tmp
Check whether there's a subfolder by xsi: /tmp/xsi/ and some numeric subfolders inside.
Temp blocks are stored in that folder and for some reason creating the tmp block is failing, why? full /tmp FS maybe?
There is some confussion around that error, we have fixed it to offer more detailed info when it is triggered. Will be available in next release.
#453 Re: General matters » Error description: can't open output file: /vmfs/volumes/... » 2021-05-28 16:06:30
You are commiting a conceptual error. Your job should read:
./xsibackup --replica "VMs(pfsense)" root@my.remote.eu:22:/vmfs/volumes/Backups/backups_exi2#454 Re: Feature requests & improvements » XSIBackup-DC 1.5 : [defaults] section in xsibackup.conf » 2021-05-27 16:14:00
Use --mail-to=mail-from as a way to shorten things. xsibackup.conf has not been thought to hold any default values, that's not the aim of the [defaults] section.
#455 Re: Feature requests & improvements » Office365 SMTP Support » 2021-05-27 08:45:52
It's applied by automatically detecting the office365 string, we might make it optional in the future.
#456 Re: Feature requests & improvements » Office365 SMTP Support » 2021-05-26 17:29:43
We added a helper in the way you suggested. Still, using services that ignore standards is not something we recommend. They arbitrarily obliterate the server in the EHLO command, why not something different tomorrow?, what kind of confidence as a user can you expect from that kind of attitude?.
#457 Re: Speed » XSIBackup-DC Perf Test » 2021-05-26 15:56:10
Of course not, we don't get you wrong. Thank you for your feedback, this is indeed the kind of talk we like.
dd just copies data in blocks from one file descriptor to another. I wonder if it uses mmap, probably not, it's a utility to slice files, a wondeful one I must say, nonetheless they probably didn't tune it for speed.
(c)XSIBackup-DC is quite optimized, we still have room for some improvement and pthread'ing is one of the improvements in our todo list. Thus, no, we haven't prioritized paralelism so far, why?
1/ Hashing & compressing data works at over 100 MB/s on an average core, given that you backup over IP and thus you don't need the server part to be using the same CPU. We call "average core" to a single thread rating of 2000 at [url=https://www.cpubenchmark.net/]CPUBenchmark[/url]. If it's a second run the processing of data happens at around 250MB/s on an average core, which is about the limit of a modern SATA HD. All of that on top of being zero aware.
2/ We are already applying parallelism since (c)XSIBackup-DC can work as a client/server.
3/ Full throttle is only required on the first pass, subsequent ones will hash and compare at 250MB/s on commodity hardware.
4/ We are now offering native CBT support, thus the above limit is simply waived by knowing which blocks have changed on advance.
5/ One of the aims of (c)XSIBackup has always beed to work in the background with a minimal footprint. Running in a single core is some good way to not create too much disturbance in a production system.
The above reasons have kept us from applying pthreads.h by now.
On the other side. Your setup is optimized for paralellism, low temp and low power consumption, still not for speed.
If you take a look at your CPU benchmark, you will see that a single core performance is below average.
[url=https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+Silver+4110+%40+2.10GHz&id=3106](c)Intel (c)Xeon Silver 4110[/url]
Just in case, remember that you must enable your smart array controller cache to obtain maximum performance in the (c)ESXi shell.
[url=https://33hops.com/hp-server-controller-cache-not-activated-issue.html]Enable your HP controller cache[/url]
Also, Windows OSs don't play well when virtualized. Microsoft has a kind of given up on the OS manufacturing niche. You probably already know this, but we'll take the chance to offer this link to an interesting post on block alignment.
[url=https://www.boche.net/blog/2009/03/20/storage-block-size-and-alignment/]VMFS and storage block alignment[/url]
When the guest FS is not aligned with the VMFS FS, this may imply that the guest needs to access more VMFS blocks to access the data it needs, if on top of that the underlying hardware's blocks aren't aligned to the VMFS ones, you can get into a situation in which the performance yielded by the system is heavily degraded.
In your case it's only the VMFS to datastore volume block alignment that matters.
#458 Re: General matters » has it updated crontab? (solved) » 2021-05-25 15:59:27
As Gundermann stated, there only one crond process, although there might me multiple jobs.
I would add that what you are doing is not very clean. You are runnign some job at 1:00 am everyday, then another one at 2:00
How do your know if first job is done when the second one is triggered?
Depending on the available resources that can cause your server to get clogged and it will surely do most of the times if your 10:00 h. job takes more than expected.
To avoid that kind of uncertainty (uncertainties is the matter Murphy's Law is made of), just chain your jobs by wrapping them in another job and call that other wrapping job instead. That way the first job will be launched and whenever it ends the second one will commence.
#459 Re: General matters » Can two XSIbackup instances use the same repository simultaneously? » 2021-05-25 15:50:35
We understood you were replicating to the same folder, as you were complaining about the lock file.
You can of course replicate to different folders, nonetheless there are some concerns regarding parallelizing streams of data that you should take into account. You will anyway end up finding it for yourself in an empirical way.
It will be much faster copying two VMs sequentially than doing it at the same time. There is a big consideration to make in respect to that when using spinning classic HDs, as there is a physical arm that has to move around to write data. If you use SSDs or nVME disks, you will work that limitation around up to some degree.
Still SCSI/SATA controllers and network equipment (if implied) will pose some limits to parallelizing streams of data and will still make serial faster than parallel most of the times.
And last but not least, replicating VMs can be a CPU intensive task. It's not just reading and writing data, (c)XSIBackup needs to hash it, compress it and look for it in hash tables in the order of millions of records. Parallelizing can take advantage of multicore systems, still you need to take that on account, as you will have to leave some CPU cycles to the rest of the system and guests.
If you want to experiment, it's O.K, share your experience here, in any case, our recommendation for a production system is that you just let (c)XSIBackup treat VMs in a row by passing all of them to the VMs() argument.
And lastly: of course, if you use --replica=cbt Changed Block Tracking feature, you will reduce data streams to the minimum, you will be parallelizing CPU usage mainly, which is OK just as long as it's OK with your server.
Use the --options=R argument to create a test VM on the other end and check your data. You can leave that VM there, even if it's turned on, the CBT feature will take care to switch it off and re-register it on each CBT cycle.
#460 Re: Speed » XSIBackup-DC Perf Test » 2021-05-25 10:56:58
It's not only the theoretical 88MB/s bottleneck. You aren't taking many other things into account:
1/ That's just a theoretical limit. Depending on your cables, switch (this has a huge impact) and NICs (also a big concern), you may get figures well below that limit.
2/ Are there any other VMs running while you do the backups?. Network bandwidth doesn't simply consist in adding up partial figures. Depending on your switch and network equipment, summing up different flows can degrade performance.
3/ SSH encryption and encapsulation require a modern CPU. Intel has SHA extensions wich will greatly enhance performance.
4/ Backing up from one non-local disk DS to another one will add up latency and require the use of the same CPU. Latency to local devices is really low on the SCSI bus. Remote attached FS will add the network latency to I/O latency.
5/ Backing up over IP will split the CPU load between client and server, still the best results will be obtained when backing up from local disks or FO to a fast NAS (SSD or nVME even better) over 10GB.
#461 Re: General matters » Installation questions. » 2021-05-24 17:04:00
The above is just a note to users coming to the post from search engines. If the software is running fine for you, let it do its work, upgrade only when you need to.
#462 Re: General matters » Longer shutdown period than 60 seconds for cold backups » 2021-05-24 12:28:45
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#463 Re: General matters » AWS SES issue w/ authentication » 2021-05-24 12:28:15
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#464 Re: General matters » Missing vdmk file .. cant start VM » 2021-05-24 12:27:42
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#465 Re: General matters » Explanation of parameter --exec is missing in man page » 2021-05-24 12:27:21
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#466 Re: General matters » Testing xsidiff in serve client mode » 2021-05-24 12:27:01
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#467 Re: General matters » --del-dir option doesn't rotate backup folder » 2021-05-24 12:26:48
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#468 Re: General matters » Backup stuck » 2021-05-24 12:26:36
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#469 Re: General matters » Copy between 2 host » 2021-05-24 12:26:19
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#470 Re: General matters » Mail report gives warning for excluded disks » 2021-05-24 12:25:48
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#471 Re: General matters » How to exclude VM's rather then include » 2021-05-24 12:25:32
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#472 Re: General matters » Chained backup problems » 2021-05-24 12:25:17
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#473 Re: General matters » Install » 2021-05-24 12:24:56
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#474 Re: General matters » Installation questions. » 2021-05-24 12:24:36
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021
#475 Re: General matters » Appear to have two running jobs at once » 2021-05-24 12:24:15
Please, note this thread corresponds to an old software edition - (c)XSIBackup Classic - which has been deprecated. Use our latest software edition: [b](c)XSIBackup-DC[/b] and refer to [url=https://33hops.com/xsibackup-dc-full-manual-home.html]its manual[/url] to [url=https://33hops.com/xsibackup-vmware-esxi-backup.html#prodchart]backup your (c)ESXi VMs[/url] as of 2021