#1 2019-01-09 10:55:35

chc-pr
Member
Registered: 2017-05-05
Posts: 24

SMTP corruption? Best way to fix + other related issues

I have just installed the pro version and thought I'd have a play with the SMTP options to try to get the scripts to work with my on-site SMTP server. Unfortunately, when I tried using TLS it failed (as I find it usually does as I have STARTTLS enabled - so its not really surprising but I thought I'd try just on the off chance). It did try to connect, so it was sort of working, so I thought I'd experiment with a few other options - all changed via the menu.

This seems to have completely killed the SMTP function. When I try to connect to anything - even using the (appropriately amended) gmail entry in the smtpconf file I still get it reporting no port is found and no server is specified - which they are.

Specifically it says (in the case of the gmail test, but it makes no difference which server it is pointed at)
Error: you set the arguement --use-smtp, but you have not configured any server. Configure the static entries for your SMTP servers in the config/smtpsrvs file or set eac KILLED
Firewall rule SMTPout-added ... is not found
Invalid Ruleset ID.
Int value is not found
SMTP probe:
^[[0;31mPlain SMTP Module ::: The email server [] is not reachable at port []^[[0m
Invalid RulesetId.
<timestamp>| Firewall rule SMTPout-closed.|

where <timestamp> is the correct timestamp of the test.
(I would post a screen grab, but I cannot see how to attach a image file)

After that, the SSH session is completely frozen.

I have tried shutting down all the VMs and rebooting ESXi, which didn't work.

Any suggestions?

On the 'Other related issues' I wanted to know;
1 - if STARTTLS support is planned and if so on what timescale?
2 - Does a complete reinstall to the same server count a second download?

Thank you for your help.

Offline

#2 2019-01-09 11:31:58

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

The conf/smtpsrvs file might have gotten corrupted somehow. It's not a big deal, as it's easy to fix, but if you managed to unformat/ corrupt it through the GUI, it could be considered a bug. It's most probably due to some special character in the input, we filter them, but you migth have passed something we are not taking into account.

Solution:
Just edit the conf/smtpsrvs file manually. The format for each line is described in the head of the file, you can just delete all active lines and start from scratch or correct manually, should you find something that should not be there.

Offline

#3 2019-01-09 12:14:56

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

I tried that already last night I'm afraid. I completely deleted the entries, then added a new one ..
1st, I tried using the menu to creat e acomplete new one
2nd, I cleared it out again, copied the sample gmail entry and uncommented it, then amended the sample to reflect my own gmail account.

Unfortunately, neither worked.

Offline

#4 2019-01-09 12:20:07

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

Here is the smtpconf entry I have

1;smtp.gmail.com:465;<myaccount>@gmail.com;<myaccount>@gmail.com;<myaccountPWD>;yes;TLS;0
2;mail2.my.domain:587;<myaccount>@my.domain;<myaccount>@my.domain;<myaccountPWD>;yes;TLS;0

with the appropriate data in the fields of course.

Offline

#5 2019-01-09 16:34:45

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

Your entry is O.K., please clarify: "Unfortunately, neither worked"
Do you mean you keep on getting?

Plain SMTP Module ::: The email server [] is not reachable at port

Why don't you just reinstall everything?, the installer will preserve your config files and RSA keys, just copy them somewhere as a backup. It'll take you just some seconds.

Please post the complete log output, so that we have all details.
Contact support, as this is complicating a bit, and looks like something particular to you.

Offline

#6 2019-01-10 08:48:55

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

admin wrote:

Your entry is O.K., please clarify: "Unfortunately, neither worked"
Do you mean you keep on getting?

Plain SMTP Module ::: The email server [] is not reachable at port

Yes, I do mean that.

admin wrote:

Why don't you just reinstall everything?, the installer will preserve your config files and RSA keys, just copy them somewhere as a backup. It'll take you just some seconds.

I'll do that. I think something has gotten corrupted somewhere.

The xsi-dir/va/logs/xsibackup.log file is empty. Is there another file I should be looking at?

Thanks for your help.

I know it is a corruption of some sort because when I was experimenting with the menu it was initially responding - and then it broke. Sorry, I didn't record my actions (I know, I know) ...

Offline

#7 2019-01-10 11:25:08

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

Well, the GUI menu has been around for some months now and we have tried all sorts of things with it. But the field of possibilities in regards to interaction sequences is endless. You don't need to record your actions, we said just in case you remembered.

The errors you are suffering are very gross, as you can see, the SMTP module is not able to retreive the SMTP server info, which is very basic functioning. A reinstall should fix it. All main files should be overwritten and your config saved.

It could be some permission issue, but I guess that's the first thing your checked.

If you feel more comfortable by doing an installation from scratch, just use this procedure to completely remove any remains:
https://33hops.com/xsibackup-uninstall-procedure.html

Offline

#8 2019-01-11 08:16:46

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

admin wrote:

Well, the GUI menu has been around for some months now and we have tried all sorts of things with it. But the field of possibilities in regards to interaction sequences is endless. You don't need to record your actions, we said just in case you remembered.

The errors you are suffering are very gross, as you can see, the SMTP module is not able to retreive the SMTP server info, which is very basic functioning. A reinstall should fix it. All main files should be overwritten and your config saved.

It could be some permission issue, but I guess that's the first thing your checked.

If you feel more comfortable by doing an installation from scratch, just use this procedure to completely remove any remains:
https://33hops.com/xsibackup-uninstall-procedure.html

Thanks. I will do that today (probably) or over the weekend. I am quite good at breaking software ... I seem to be able to find bugs no-one ever encounters ...

Still VERY happy with this product. Been using the free version for a couple of months and wanted to make use of some of the pro features (and wanted to support your work too), but I like to mess about with things and see what is possible before relying on it ... just curiousity really.

I very much appreciate the service you are providing. Thank you.

Oh and yes, there is no permission issue ... I have had it working earlier and then broke it by experimenting with settings :-)

One outstanding question, is STARTTLS on the horizon at all?

Last edited by chc-pr (2019-01-11 08:18:57)

Offline

#9 2019-01-11 18:37:56

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

Well, I don't have the newer add ons to the SMTP spec now in mind, but, isn't that a feature on the server side upgrading non secure SMTP to a secured connection?

XSIBackup does indeed support TLS through OpenSSL if the server does.

Offline

#10 2019-01-14 08:11:38

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

admin wrote:

Well, I don't have the newer add ons to the SMTP spec now in mind, but, isn't that a feature on the server side upgrading non secure SMTP to a secured connection?

XSIBackup does indeed support TLS through OpenSSL if the server does.

I believe that STARTTLS is a little different to TLS, not too sure how, but everytime I have tried to use TLS to connect to the STARTTLS port it fails with pretty much every client I have tried. I have never had a STARTTLS option have any difficulty.

I don't actually know what the difference is, but there clearly is some difference.

Offline

#11 2019-01-14 13:11:01

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

STARTTLS is a mechanism by which an SMTP communication that started as non-secure is turned into a TLS session.

Offline

#12 2019-01-14 14:02:09

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

That was my understanding ... but it doesn't seem to talk to TLS properly. Maybe its a bug somewhere in my mailserver.

Anyway, more pressing is that the issue with sending any emails and the menu is still not resolved with a complete clearout and reinstall. More confusing still is that the gmail account I was using is not working either since the move to the Pro version.

I have copies of some test scripts I used with the old free version that were proven to work fine. I cut and pasted this into a SSH session (as I had done successfully with the old 'free' install) and that worked (ie it did backup the VM apparently) right up until is was time to email the report when it crashed.

Thinking that maybe something had changed at gmail I tried the same script (modified to accommodate a different VM resident on that server) on another ESXi with the XSIbackup-free version install I have (I kept that as it was working just in case) and that worked flawlessly.

The only difference is one is ESXi 6.5, the other (non-working one) 6.7 and a recent new install.

I think it may indeed be some permission somewhere, but I cannot see what or how as the install and SSH session are all as root. I think it might be a firewall issue ....

I don't want to do this, but maybe I need to uninstall fully again and reinstall the free version on the non-working ESXi and see if teh free version still works ... unless you have any other ideas ...

Last edited by chc-pr (2019-01-14 14:07:04)

Offline

#13 2019-01-14 16:32:50

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

We ourselves use GMail and other e-mail accounts without issue. There aren't any open issues around this problem either.

1/ Remember that XSIBackup SMTP Client requires the PIPELINING extension enabled in your SMTP server.
2/ Remember to enable Less Secure Apps in your GMail account.
3/ Sometimes, you may have trouble with timing in between SMTP lines, in case of GMail this is by default, as they require a delay in between commands, which is already built in the XSIBackup SMTP client. You have the --smtp-delay argument, which you can configure in your conf/smtpsrvs file, it's the last value, which is by default set to zero (0).
4/ You have the following variables in the src/sendmail file.

    SMTP_CHECK_INTERVAL="1"
    SMTP_CHECK_TIMEOUT="7"

Which control some timeouts when probing the SMTP servers prior to sending the e-mail.

But please, do take on account that the last detail you posed about this issue:

Plain SMTP Module ::: The email server [] is not reachable at port

Stated that the SMTP module wasn't even able to know what server it had to contact. Please note the empty space in between the brackets.

Offline

#14 2019-01-15 08:58:35

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

Thank you for taking the time to respond yet again. I do appreciate it.

I think maybe I have been less clear about where I am with this issue than I should have been, so I am going to clarify the current position in detail. Please assume unless it is included here that the afore mentioned is not happening any more.

You can take it that all actions were undertaken as 'root'. I am the only one with physical access to this server and it is entirely blocked by a firewall from the outside world, so I didn't bother setting up any other users. I also disable SSH (even though port 22 is blocked by the external firewall) except when I want to access the server myself - when I manage it locally and turn port 22 on - then conduct all work on teh server using a 'Putty' SSH session.

Summary of case
1) Installed XSIbackup-free version - worked perfectly
2) deleted XSIbackup-free version and installed XSIbackup-pro version. Initially seemed to work (I did not do a backup - only tested the email setup menu) but was unable to get a connection to my local mail server. Tried a few things but no success, then it suddenly stopped working hanging with the messages described earlier
3) Tried rebooting the ESXi server - didn't help
4) Uninstalled the XSIbackup-pro version and reinstalled it. Menu now seems to work again, but still not establishing a connection to my local server so I then reverted to using the gmail account (by manually editing the smtpsrvs file copying the REM'd gmail example to use my details) I had been using for the XSIbackup-free test install. I was still unable to establish a connection.

I then went back to the test script I was using with the XSIbackup-free version (which included the gmails details) and ran that. Once again, I failed to establish a connection to gmail.

I took the EXACT same script which failed to connect to gmail and changed only the references to the VM to backup (keeping the same backup destination path) and ran that on another ESXI server which still has a copy of XSIbackup-free and that script executed fully and perfectly including sending the email via gmail.

I can therefore positively eliminate any issue with the gmail account end.

The EXSi implementation on which XSIbackup-Pro is installed is version 6.7. The other install (XSIbackup-Free) is on a ESXi version 6.5 box. The 6.7 version worked perfectly with the original 'free' install. I guess it is possible that the issue is a corruption of some sort within the ESXi install itself, but I would rather not wipe the install and start again unless I have to ... obviously.

This is from my last test this AM using the new pro version installation ...

No errors detected in backup
-----------------------------------------------------------------------------------------
Int value is not found
Firewall rule SMTPout-465 added...
Invalid Ruleset Id.
Int value is not found
SMTP probe:
GMail Module ::: The e-mail server [smtp.gmail.com] is not reachable at port [465]
Invalid Ruleset Id.
Firewall rule SMTPout-465 closed.
Backup finished

So some progress of sorts.
When I test the gmail settings using the menu I get the following before it hangs and I have to issue ^C to get the prompt back (which is also some progress on past days before I reinstalled as previously ^C was not responsive either.

lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
x Checking server smtp.gmail.com:465...                                                 x
xqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqx
x Using stored SMTP server info...                                                      x
x Found conf/smtpsrvs file...                                                           x
x Int value is not found                                                                x
x Firewall rule SMTPout-465 added...                                                    x
x Invalid Ruleset Id.                                                                   x
x Int value is not found                                                                x
x SMTP probe:                                                                           x
x ^[[0;31mGMail Module ::: The e-mail server [smtp.gmail.com] is not reachable at port [x
x Invalid Ruleset Id.                                                                   x
x 2019-01-15T08:34:29|  Firewall rule SMTPout-465 closed.  

Clearly, something is missing.

I have also triple checked your last suggestions re smtpsrvs and sendmail files and both checked out OK.

I am kind of left scratching my head ...

Last edited by chc-pr (2019-01-15 09:00:43)

Offline

#15 2019-01-15 13:13:33

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

You are ignoring some fundamental information being returned by (c)XSIBackup:

Int value is not found
Firewall rule SMTPout-465 added...
Invalid Ruleset Id.
Int value is not found

XSIBackup is not being able to open the 465 port in the ESXi firewall.

chmod 644 /etc/vmware/firewall/service.xml
vi /etc/vmware/firewall/service.xml

Check at the end of the file whether there's any garbage or half written rute and delete it.
You can of course always manually add the rule via the vSphere client
Or, overwrite the /etc/vmware/firewall/service.xml file with some default one from other server.

Offline

#16 2019-01-15 13:21:21

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

The exact way the XSIBackup firewall management function works is:

1 - It issues esxcli network firewall ruleset list in the command line and checks whether some rule called SMTPout-465 does already exist.
2 - If it exists, it enables it by issuing esxcli network firewall ruleset set --ruleset-id="SMTPout-465" --enabled=true.
3 - If it does not exist, it adds it to the firewall file /etc/vmware/firewall/service.xml

Offline

#17 2019-01-16 09:12:13

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

I think I may have found something. I issued the two commands that you mentioned above. I could see no ref to SMTPout-465 in ruleset list, but I did find MULTIPLE references to that rule id at the end of the config file - all with <rule id='0000'> set as shown below in the partial code snippet.  There is at least two full scroll pages of this, I only included what I could get on one screen grab of text easily ... there is much more of the same.

        <porttype>dst</porttype>
        <port>465</port>
     </rule>
     <enabled>true</enabled>
     <required>false</required>
  </service>
   <service id='9999'>
     <id>SMTPout-</id>
     <rule id='0000'>
        <direction>outbound</direction>
        <protocol>tcp</protocol>
        <porttype>dst</porttype>
        <port></port>
     </rule>
     <enabled>true</enabled>
     <required>false</required>
  </service>
   <service id='9999'>
     <id>SMTPout-465</id>
     <rule id='0000'>
        <direction>outbound</direction>
        <protocol>tcp</protocol>
        <porttype>dst</porttype>
        <port>465</port>
     </rule>
     <enabled>true</enabled>
     <required>false</required>
  </service>
   <service id='9999'>
     <id>SMTPout-465</id>
     <rule id='0000'>
        <direction>outbound</direction>
        <protocol>tcp</protocol>
        <porttype>dst</porttype>
        <port>465</port>
     </rule>
     <enabled>true</enabled>
     <required>false</required>
  </service>
   <service id='9999'>
     <id>SMTPout-465</id>
     <rule id='0000'>
        <direction>outbound</direction>
        <protocol>tcp</protocol>
        <porttype>dst</porttype>
        <port>465</port>
     </rule>
     <enabled>true</enabled>
     <required>false</required>
  </service>
   <service id='9999'>
     <id>SMTPout-465</id>
     <rule id='0000'>
        <direction>outbound</direction>
        <protocol>tcp</protocol>
        <porttype>dst</porttype>
        <port>465</port>
     </rule>
     <enabled>true</enabled>
     <required>false</required>
  </service>
 </ConfigRoot>

Should I just delete all these references and try again?

I should add that I also noted that at least one of the repetitions has no port number listed ...

   <service id='9999'>
     <id>SMTPout-</id>
     <rule id='0000'>
        <direction>outbound</direction>
        <protocol>tcp</protocol>
        <porttype>dst</porttype>
        <port></port>
     </rule>
     <enabled>true</enabled>
     <required>false</required>
  </service>

I bet that doesn't help either, especially as they all have the same service and rule id's

I await your advice with interest. Thank you for this help.

Last edited by chc-pr (2019-01-16 09:12:43)

Offline

#18 2019-01-16 13:11:28

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

Yes, you can delete all those or just start with a default firewall template, whatever you find easier to accomplish.
We have reinforced checks to prevent ghost rules from being added, we never had such case though, if fact there already existed checks around this.

You seem to have passed some SMTP configuration without a port, that generated partial FW rules which were not being detected, so every time you tried, a new one was being generated.

Offline

#19 2019-01-17 11:45:16

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

:-) I strike again :-)

I'll delete and get back to you to confirm it sorted the issue ... or not as the case may be. Thanks again.

Offline

#20 2019-01-17 12:49:12

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

I have now deleted the offending Port references, rebooted the ESXi server to make sure that the revised firewall is properly loaded, but it is basically back to doing what it was when I first posted here namely ...

Error: you set the arguement --use-smtp, but you have not configured any server. 
Configure the static entries for your SMTP servers in the config/smtpsrvs file or set eac 
KILLED
Firewall rule SMTPout-added ... is not found
Invalid Ruleset ID.
Int value is not found
SMTP probe:
^[[0;31mPlain SMTP Module ::: The email server [] is not reachable at port []^[[0m
Invalid RulesetId.
<timestamp>| Firewall rule SMTPout-closed.

Offline

#21 2019-01-18 13:03:48

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

You have some syntax error in your conf/smtpsrvs file, please contact support and submit the contents of the file, whithout passwords of course.

Offline

#22 2019-01-21 11:57:14

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

OK, Will do it today. Thanks

Offline

#23 2019-01-31 19:25:23

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

This is a problem restricted to sending e-mails, so let’s concentrate in this issue and leave everything else apart.
Issue this command:

./xsibackup --check-smtp=some@email.com --use-smtp=2

And let us know if you get the same result, I bet you will.
This looks like an ESXi bug in some build when calling Firewall operations from the command line or a broken firewall more than an XSIBackup issue.
Try adding the firewall command manually at the end of the /etc/vmware/firewall/service.xml file…

<service id='9999'>
    <id>SMTPout-587</id>
    <rule id='0000'>
        <direction>outbound</direction>
        <protocol>tcp</protocol>
        <porttype>dst</porttype>
        <port>587</port>
    </rule>
    <enabled>true</enabled>
    <required>false</required>
</service>

And see what is ESXi’s response when issuing this command

esxcli network firewall ruleset set --ruleset-id="SMTPout-587”--enabled=true

Offline

#24 2019-02-04 11:10:05

chc-pr
Member
Registered: 2017-05-05
Posts: 24

Re: SMTP corruption? Best way to fix + other related issues

Thank you, I really appreciate this suggestion.

Running ./xsibackup --check-smtp=some@email.com --use-smtp=2 yields ...

Using stored SMTP server info...
Found conf/smtpsrvs file...
Int value is not found
Firewall rule SMTPout-465 added...
Invalid Ruleset Id.
Int value is not found
SMTP probe:
GMail Module ::: The e-mail server [smtp.gmail.com] is not reachable at port [465]
Invalid Ruleset Id.
Firewall rule SMTPout-465 closed.
Killed
[root@fxcn4:/vmfs/volumes/5beaacc7-2506215d-5adb-300ed500100e/xsi-dir]

As for

esxcli network firewall ruleset set --ruleset-id="SMTPout-587”--enabled=true 

I just get a

>

prompt from which I cannot get out. I tried ctrl-c, quit, q, Q, ctrl-q, ; and : ... I can't think of anything else which might work.

I should perhaps add that there were again multiple instances of the SMTPout456 service added again having previously cleared them out.

Offline

#25 2019-02-04 20:36:51

admin
Administrator
Registered: 2017-04-21
Posts: 1,363

Re: SMTP corruption? Best way to fix + other related issues

To get out of that prompt, which is the bash continuation, type Ctrl+d
You are pasting some unknown character, type the command manually to make sure that you are entering the right characters.

You must first add the rule

<service id='9999'>
    <id>SMTPout-587</id>
    <rule id='0000'>
        <direction>outbound</direction>
        <protocol>tcp</protocol>
        <porttype>dst</porttype>
        <port>587</port>
    </rule>
    <enabled>true</enabled>
    <required>false</required>
</service>

To the rules file at /etc/vmware/firewall/service.xml
Once you have done so

esxcli network firewall ruleset set --ruleset-id="SMTPout-587”--enabled=true

Offline

Board footer