#1 2017-10-16 19:01:52

d4t1983
Member
Registered: 2017-10-16
Posts: 2

How to exclude VM's rather then include

Hi,

Is it possible to exclude VM's based on a regexp or similar?

I want to backup all my VM's except for a select few, just trying to find the easiest way.

Thanks in advance.

Offline

#2 2017-10-17 14:02:55

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

Just a few ideas, you can:

1/ Select them manually: --backup-type=custom --backup-vms=VM1,VM2, etc...
2/ Use a regex to select/deselect VMs (available from v. 10.0.4, will be launched in a few days)
3/ Script the VM list in an include source file within xsibackup-cron and use the variable in place of the VM list.

Offline

#3 2017-10-17 14:47:36

d4t1983
Member
Registered: 2017-10-16
Posts: 2

Re: How to exclude VM's rather then include

Thanks for that, in the end I decided to wrap xsibackup inside another script

Offline

#4 2018-05-08 04:39:47

balu
Member
Registered: 2018-04-01
Posts: 6

Re: How to exclude VM's rather then include

It would be nice to have an exclude option for a whole vms and not only disks inside vm-s.

--backup-type=all \
--backup-vms="!Exclude-This-VM,!AlsoExcludeThisVM" \


Could you provide an exclude regex example for me?

Thanks!

Offline

#5 2018-05-08 08:26:42

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

You can select VM by means of a Regular Expression as described in the product's man page: https://33hops.com/xsibackup-help-man-p … pvmsregexp

Offline

#6 2018-05-08 12:26:36

balu
Member
Registered: 2018-04-01
Posts: 6

Re: How to exclude VM's rather then include

admin wrote:

You can select VM by means of a Regular Expression as described in the product's man page: https://33hops.com/xsibackup-help-man-p … pvmsregexp

I would like to have a job that backups all the VM's (even the newly created ones) and exclude a few one (for example the ones that contain the keyword "DVR" in their name)

I do not think that this can be done with REGEX, as I'm unable to exclude that way. Am I right, or I'm missing something?

Thanks!

Offline

#7 2018-05-09 14:02:06

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

Off course it can be done with REGEX. You just need to create your matching expression.

Learning Regular Expressions is easy, but as everything it takes some time and a learning curve.
You can start here: https://regexone.com/

Offline

#8 2018-05-15 08:44:16

tothand
Member
Registered: 2018-05-10
Posts: 3

Re: How to exclude VM's rather then include

admin wrote:

Off course it can be done with REGEX. You just need to create your matching expression.

Learning Regular Expressions is easy, but as everything it takes some time and a learning curve.
You can start here: https://regexone.com/

The documentation says

Select VMs using REGEXP:

Basic regular expressions are allowed to select VMs. The regexp engine in use is that of grep utility built into Busybox, in turn bundled with ESXi, thus if you want to practice expressions before you use them in your --backup-vms argument, you can just use this one-liner.

I'd like to point out the part  Basic regular expressions. In order to do an exclusion you would need a negative lookahead regex pattern e.g:

^(?!.*something)

And this is not part of the basic regular expression.

So unless all of us in the thread are missing something here the exclusion can't be done with regex since there is no argument like --exclude-vms="...."

The only solution I can think of is to create a vmlist in a wrapper shell script like this:
(offtopic: The awk filter below only works if there are no spaces in the VM names which is the case for me)

VMLIST=`vim-cmd vmsvc/getallvms | awk '!/Vmid/ && !/something/{print $2}'`

And then use it like this

xsibackup ... --backup-vms="$VMLIST" ...

This is basically the 3rd option you've mentioned in your first reply, but this complicates things when used with --host because then we need to gather the VM list through SSH from the remote host so the clean solution would be an official support for exclusion wither via a separate argument like --exclude-vms="..." or a modifier like --backup-vms="!skipthis,!skipthat,includethis".
My bet is on the former (--exclude-vms) since the exclamation mark already used in backup-vms to exclude certain disks from the backup.

So it seems that the feature request is legit because right now it's not possible to do a proper exclusion there are only workarounds which are creating further problems.

I'm not a regex guru myself so it is possible that there is a pattern which solves the issue. If you know it please share.

Thanks

On a personal note: I believe pointing someone to a regular expression tutorial site saying 'of course it can be done' when neither the documentation nor you have provided a working solution for the question is not cool even if you're right and there is a pattern which solves the issue.

Offline

#9 2018-05-17 14:06:00

bpreston
Member
Registered: 2018-05-15
Posts: 5

Re: How to exclude VM's rather then include

I agree with the sentiment--having an exclude VM option would be handy.  It would be nice to have the software backup all VMs that it finds except ones matching certain parameters.  The "disks to backup" syntax has negative(exclusion) parameters, so it does seem odd that --backup-vms only uses positive(include) parameters.


Apologies if you already know this, but if your VM hosts (that you want backed up) are relatively static*, you can send the entire list of VMs (which you do want backed up) in the commandline (or save it in your VMLIST value and import that).  xsibackup will only backup the ones that it finds on that particular host, ignoring those not found.


*=I know you said "newly created ones", so I guess your implication is that you have an environment where there are VMs being brought up routinely, and those new VMs do need to be backed up.  Could you perhaps control the naming of those new VMs being brought up?  So, say, it is known in your environment that any VMs brought up with a name starting with "Prod" will be backed up automatically? You would then feed that regex string to --backup-vms?

Offline

#10 2018-05-17 15:43:20

tothand
Member
Registered: 2018-05-10
Posts: 3

Re: How to exclude VM's rather then include

bpreston wrote:

Apologies if you already know this, but if your VM hosts (that you want backed up) are relatively static*, you can send the entire list of VMs (which you do want backed up) in the commandline (or save it in your VMLIST value and import that).

The problem (complication) with this approach is that if you use a shell variable to do this, you can't send this variable to other host with the --host parameter. (Maybe you can, but I haven't figured it out yet)
Probably because the variable name gets to the other host not the content so you would need to maintain a list of VMs to be backed up from all your hosts in one place where the backup cron runs.
Sure, it can be done just as I wrote in my latest post

tothand wrote:

this complicates things when used with --host because then we need to gather the VM list through SSH from the remote host

The idea to prefix the "important" VMs with some string is nice but if you have a few hosts with a couple of dozen VMs already in place which doesn't follow this naming convention then you have a lot of extra work on your hands to rename them as needed.
(The problem is not the renaming but the whole administrative burden, e.g. the filenames won't match the VM name and if you have some custom scripts using the VM names for something those will have to be modified too.)

Or another approach is to have a static list of the VMs already in place and needed to be backed up AND having a unique prefix for the new VMs needed to be backed up so you end up with a command line like this:

xsibackup ... --backup-vms="vm1, vm2, vm3, REGEX(^prod_)" ...

So there are workarounds but none of them offers a clean, easily maintainable, unified solution.

Like you said:

bpreston wrote:

he "disks to backup" syntax has negative(exclusion) parameters, so it does seem odd that --backup-vms only uses positive(include) parameters.

By the way my problem is not that it can't be done but that no one willing to confirm this from the devteam and say "Sorry, it can't be done right now, maybe we will consider this in a future version" and then I won't have to bang my head against the wall how to do it or why I haven't figured it out yet smile

Offline

#11 2018-05-17 16:14:55

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

We are in the middle of big changes, so sorry if we are not offering you the reactivity you expect.
XSIBackup-Pro 11 will change from jobs stored in the xsibackup-cron file to job files stored in the /jobs directory. This will open up new doors to being able to exchange information between hosts in a much more convenient way, and will wipe away all problems related to sending invidual jobs as a string through an SSH tunnel, related to escaping and so on.
On addition XSIBackup-Pro 11 will use the ESXi crontab, so it will be as flexible as a regular Linux crontab is, and much more ;-)

Excluding disks is an excluding feature, while selecting VMs through a REGEXP is an inclusive feature.

In regards to selecting VMs. We are talking about dozens of items, hundreds at most. The simple grep REGEXP functionality should be enough to handle VM selection. This is when having created the VM names in base to an algorithm (something simple, no rocket science) comes in handy. If you have fixed length names with intrinsic meaning, i.e: WINACC001, LINADM002, etc... using a REGEXP is easy. If your naming does not follow a convention, it doesnt matter how powerful your REGEXP tool is, you will be doomed to almost a one by one basis picking.

XSIBackup is flexible enough to offer you many different ways to approach a problem like this.

In any case, please, write to support with your problem's details and we'll give it a thought and publish it here for others to take advantage too.

Offline

#12 2018-05-17 16:24:24

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

tothand wrote:
bpreston wrote:

Apologies if you already know this, but if your VM hosts (that you want backed up) are relatively static*, you can send the entire list of VMs (which you do want backed up) in the commandline (or save it in your VMLIST value and import that).

The problem (complication) with this approach is that if you use a shell variable to do this, you can't send this variable to other host with the --host parameter. (Maybe you can, but I haven't figured it out yet)
Probably because the variable name gets to the other host not the content so you would need to maintain a list of VMs to be backed up from all your hosts in one place where the backup cron runs.
Sure, it can be done just as I wrote in my latest post

tothand wrote:

this complicates things when used with --host because then we need to gather the VM list through SSH from the remote host

The idea to prefix the "important" VMs with some string is nice but if you have a few hosts with a couple of dozen VMs already in place which doesn't follow this naming convention then you have a lot of extra work on your hands to rename them as needed.
(The problem is not the renaming but the whole administrative burden, e.g. the filenames won't match the VM name and if you have some custom scripts using the VM names for something those will have to be modified too.)

Or another approach is to have a static list of the VMs already in place and needed to be backed up AND having a unique prefix for the new VMs needed to be backed up so you end up with a command line like this:

xsibackup ... --backup-vms="vm1, vm2, vm3, REGEX(^prod_)" ...

So there are workarounds but none of them offers a clean, easily maintainable, unified solution.

Like you said:

bpreston wrote:

he "disks to backup" syntax has negative(exclusion) parameters, so it does seem odd that --backup-vms only uses positive(include) parameters.

By the way my problem is not that it can't be done but that no one willing to confirm this from the devteam and say "Sorry, it can't be done right now, maybe we will consider this in a future version" and then I won't have to bang my head against the wall how to do it or why I haven't figured it out yet smile

We do like exigent clients and users, but I think you are going faster than we are. XSIBackup is probably the most flexible backup tool in the market, but you are taking the matter to limits that are more in the custom development zone, than in the user program's

Offline

#13 2018-05-17 16:38:02

tothand
Member
Registered: 2018-05-10
Posts: 3

Re: How to exclude VM's rather then include

Well, great news, thanks for sharing!

As for the REGEXP, you're right, if the names are selected with this feature in mind there is no problem but for us it's a bit of a hiccup because we're carrying a lot of legacy stuff and we only know XSIBackup for a few months (awesome tool by the way!) so we were not prepared for this naming convention.

Anyway, thanks for the info, we're looking forward to see what does version 11 can offer!

admin wrote:

but you are taking the matter to limits that are more in the custom development zone, than in the user program's

Well, as a developer myself I usually test the limits - sometimes unintentionally - of the softwares, sorry about that but hey this is how software evolves smile

XSIBackup is a great product and we really like it, just sometimes we hit a bump in the road and we're trying to figure a way around and this forum is a good place to start because people are throwing in ideas, different approaches.

I hope the community around it will grow further!

Offline

#14 2018-05-17 17:35:51

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

Thank you for the positive comments and feedback.

We believe the concept can be stretched way further than where we are now. In any case it's becoming harder to find people willing to pay for a "black screen" solution. People in general believe to find more value in a half featured multicolor, semi-transparent, window based application than in more serious software. Virtualization has not only taken over computing science, but people's minds, nobody wants to face the truth in front of an SSH terminal, but just a few brave nerds :-), most prefer the blue pill and click here and there to see if they can get to achieve what they wanted to.

We are nevertheless aware of the fact that we need to make XSIBackup easier to use, which in the end will make it faster and more reliable. Next main release, which we are finishing now, will contain a GUI and will be more integrated with ESXi, will use it's crontab and manage jobs from a file level perspective, allowing to develop new concepts, like distributing full projects in packages and a lot more.

Offline

#15 2019-03-26 21:18:09

NextLevel
Member
Registered: 2019-03-11
Posts: 33

Re: How to exclude VM's rather then include

Still no VM exclude option on the road map  ? Like "All running, but not "*_temp" ?  Would still be really helpful ...

Last edited by NextLevel (2019-03-26 21:32:46)

Offline

#16 2019-03-27 15:34:57

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

You don't need an exclude feature when you have REGEXP at your disposal. XSIBackup uses grep -E functionality, which offers extended REGEXP support, including grouping

https://www.digitalocean.com/community/ … x#grouping

And negation

https://stackoverflow.com/questions/104 … ing-string

So, practice your regexp in your ESXi command line and then paste it to your job.

Offline

#17 2019-03-27 22:56:58

NextLevel
Member
Registered: 2019-03-11
Posts: 33

Re: How to exclude VM's rather then include

Sure: Why easy, if complicated. wink

RegExp always will often fry your brain ...

Just putting a pattern as REGEXP() in the --backup-vms option ... OK. But you can not define omission in this way. Isn't it?

But "greping" around... with some command constructions where you can do more wrong than right?

Well, we have to live with it. Anyway, please put it on the feature request list.

Last edited by NextLevel (2019-03-28 14:37:29)

Offline

#18 2019-04-03 01:06:00

NextLevel
Member
Registered: 2019-03-11
Posts: 33

Re: How to exclude VM's rather then include

Since there is yet no built-in way to backup all running VMs while excluding some, I made this script:

OLDIFS="$IFS"
IFS=$'\n'
VMList=$(vim-cmd vmsvc/getallvms | sed '1d' | awk '{if ($1 > 0) print $1" "$2}')

for VM in $VMList; do

    VMName=$(echo $VM | awk '{print $2}')
    VMID=$(echo $VM | awk  '{print $1}')

    VMMatch=false
    for pattern in $@; do
        if (echo $VMName | grep -q $pattern)
        then  VMMatch=true; break
        fi
    done
    if $VMMatch; then continue; fi

    if (vim-cmd vmsvc/power.getstate $VMID | grep -q "Powered on")
    then VMsToBackup="$VMsToBackup,$VMName"
    fi
done

IFS=$OLDIFS
echo ${VMsToBackup#?}

Save it with the name AllRunningButNot then you can use it in the --backup-vms option with one or more exclude patterns as parameter:

--backup-vms=$(./{path_where_you_store_it}/AllRunningButNot .*temp test ^clone.*)

In the example above, all running VMs are selected, unless the name ends with "temp", contains the string "test" or starts with "clone"

I only use underscore, hyphen and period as special characters in my VM names. I don't know if it works with others.

The bad thing of this workaround is, you cannot use the PROs GUI to manage the job.

Offline

#19 2019-04-03 11:49:18

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

The GUI has been designed to speed up first jobs creation. Once you have some basic jobs set up, you will probably find it a lot easier to just duplicate jobs and edit them using vi.

Your script is a good example of how you can extend XSIBackup functionality, but, isn't it a bit of crossing the river to get water?. Don't you think it'd be a lot easier to write a REGEXP with an exclussion group instead of adding an external module.

Offline

#20 2019-04-03 14:02:38

NextLevel
Member
Registered: 2019-03-11
Posts: 33

Re: How to exclude VM's rather then include

I've  tried, but as you say you can't use it in combination with --backup-type=Running, so I had no choice

But even if I ignore the power state, I do not have a clue what a REGEXP () pattern should look like to achieve the same exclusions as described.

Any suggestion is welcome

Last edited by NextLevel (2019-04-03 14:08:24)

Offline

#21 2019-04-03 14:07:34

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

--backup-type=custom --backup-vms=REGEXP

Offline

#22 2019-04-03 14:09:24

NextLevel
Member
Registered: 2019-03-11
Posts: 33

Re: How to exclude VM's rather then include

How to exclude .*temp test ^clone.* ?

Offline

#23 2019-04-03 14:21:35

admin
Administrator
Registered: 2017-04-21
Posts: 1,367

Re: How to exclude VM's rather then include

We may some day dedicate a post to REGEXP patterns, but by now, it's a general purpose tecnology which is well stablished and well documented in a multitude of websites, i.e.:
https://www.digitalocean.com/community/ … s-in-linux

Offline

#24 2019-04-03 14:36:54

NextLevel
Member
Registered: 2019-03-11
Posts: 33

Re: How to exclude VM's rather then include

Just say how to implement the necessary -v option of grep into the --backup-vms=REGEXP   big_smile

Because something like --backup-vms="REGEXP('^(?!.*temp).*$')"

simply does not work

Last edited by NextLevel (2019-04-03 15:43:17)

Offline

#25 2020-07-04 11:44:18

sphen
Member
Registered: 2020-06-13
Posts: 45

Re: How to exclude VM's rather then include

I have modified the given script to account for VMs that may have a space in the name:

OLDIFS="$IFS"
IFS=$'\n'
VMList=$(vim-cmd vmsvc/getallvms | sed '1d' | awk -F' {2,}' '{if ($1 > 0) print $1" "$2}')

for VM in $VMList; do

    VMName=$(echo $VM | awk '{$1=""; print $0}' | awk '{$1=$1};1')
    VMID=$(echo $VM | awk  '{print $1}')

    VMMatch=false
    for pattern in $@; do
        if (echo $VMName | grep -q $pattern)
        then  VMMatch=true; break
        fi
    done
    if $VMMatch; then continue; fi

    if (vim-cmd vmsvc/power.getstate $VMID | grep -q "Powered on")
    then VMsToBackup="$VMsToBackup,$VMName"
    fi
done

IFS=$OLDIFS
echo ${VMsToBackup#?}

Offline

Board footer