F5 SMTP and SNMP Trap Alert Issues

 I ran across an F5 cluster that had issues with sending syslogs to our internal mail relay in order to get these logs via email. There were several issues I found.

First, test your SMTP and SNMP Traps config:
SMTP:

echo "SMTP Test Email" | mail -vs "Test email for KC" kcordero@cordero.me

SNMP Traps:

logger -p local0.notice "01070638:5: Pool /Common/pool_one member /Common/192.168.10.1:80 monitor status down."
logger -p local0.notice "01070727:5: Pool /Common/me member /Common/192.168.10.1:80 monitor status up."

logger -p local0.notice "01070640:5: Node 1.1.1.1 monitor status down."
logger -p local0.notice "01070728:5: Node 1.1.1.1 monitor status up."

For me both were not working so I had to get the SMTP working first because I was getting an error. Below is the error I was getting:

[admin@F5-bigip1:Active:In Sync] ~ # echo "SMTP Test Email" | mail -vs "Test email for KC" kcordero@cordero.me
send-mail: Cannot open localhost:25

Here’s the error in the log:

[admin@F5-bigip1:Active:In Sync] ~ # cat /var/log/maillog
Jan  4 09:52:45 F5-bigip1 err sSMTP[10482]: Unable to connect to "localhost" port 25.
Jan  4 09:52:45 F5-bigip1 err sSMTP[10482]: Cannot open localhost:25
Jan  4 12:24:01 F5-bigip1 err sSMTP[14036]: Unable to connect to "localhost" port 25.
Jan  4 12:24:01 F5-bigip1 err sSMTP[14036]: Cannot open localhost:25
Jan  4 12:33:40 F5-bigip1 err sSMTP[15968]: Unable to connect to "localhost" port 25.
Jan  4 12:33:40 F5-bigip1 err sSMTP[15968]: Cannot open localhost:25
Jan  4 12:33:50 F5-bigip1 err sSMTP[15993]: Unable to connect to "localhost" port 25.
Jan  4 12:33:50 F5-bigip1 err sSMTP[15993]: Cannot open localhost:25

So then I looked at the SMTP.CONF file:

[root@F5-bigip1:Active:In Sync] config # cat /etc/ssmtp/ssmtp.conf
#
# THIS IS AN AUTO-GENERATED FILE - DO NOT EDIT!!!
#
#
# /etc/ssmtp.conf -- a config file for sSMTP sendmail.
#
# See the ssmtp.conf(5) man page for a more verbose explanation of the
# available options.
#

# The person who gets all mail for userids < 500
# Make this empty to disable rewriting.
root=postmaster

# The place where the mail goes. The actual machine name is required
# no MX records are consulted. Commonly mailhosts are named mail.domain.com
# The example will fit if you are in domain.com and your mailhub is so named.
mailhub=mail.cordero.me:25

# Example for SMTP port number 2525
# mailhub=mail.your.domain:2525
# Example for SMTP port number 25 (Standard/RFC)
# mailhub=mail.your.domain
# Example for SSL encrypted connection
# mailhub=mail.your.domain:465

# Where will the mail seem to come from?
# RewriteDomain=

# The full hostname
# Hostname=

# Set this to never rewrite the "From:" line (unless not given) and to
# use that address in the "from line" of the envelope.
FromLineOverride=YES

# Use SSL/TLS to send secure messages to server.
#UseTLS=YES

# Use SSL/TLS certificate to authenticate against smtp host.
#UseTLSCert=YES

# Use this RSA certificate.
#TLSCert=/etc/pki/tls/private/ssmtp.pem

# Get enhanced (*really* enhanced) debugging information in the logs
# If you want to have debugging of the config file parsing, move this option
# to the top of the config file and uncomment

I noticed the "mailhub=" what commented out. At the same time, I removed the comment for "FromLineOverride=" because I want to override this.

After this, I was getting my test email alerts. So now onto the SNMP Traps. An important one for this is to make sure your OID & alert name matches the OID & name that's in "cat /etc/alertd/alert.conf" or else it won't work. Below is my "alert.conf" settings and a PIC of the "alert.conf" you need to match. Notice the OID & name in there. They need to match. The consultant who set this before me created them with different names. You can do that if their custom and your not using the default values.

My alert.conf config:

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.10";
    email toaddress="network@cordero.me"
    fromaddress="F5@cordero.me"
    body="Pool Member Down!"
}

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS_UP {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.11";
    email toaddress="network@cordero.me"
    fromaddress="F5@cordero.me"
    body="Pool Member Up!"
}

alert BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.12";
    email toaddress="network@cordero.me"
    fromaddress="F5@cordero.me"
    body="Node is Down!"
}

alert BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS_UP {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.13";
    email toaddress="network@cordero.me"
    fromaddress="F5@cordero.me"
    body="Node is Up!"
}

Screenshot of the "alert.conf" settings:

Another thing I had to do was to make sure MAILTO="" is at the top of the outputs below. Mine where NOT at the top and I had to move them. This was common if you upgraded your F5's software version.

crontab -e -u root

MAILTO=""
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
HOME=/

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

cat /etc/crontab

MAILTO=""
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
HOME=/

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

cat /etc/cron.d/0hourly

MAILTO=""
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
HOME=/
01 * * * * root run-parts /etc/cron.hourly

Below I just checked to see if the "MAILTO=" didn't have anything after the =.

grep -i mailto /etc/cron*

[admin@F5-bigip1:Active:In Sync] ~ # grep -i mailto /etc/cron*
grep: /etc/cron.d: Is a directory
grep: /etc/cron.daily: Is a directory
grep: /etc/cron.hourly: Is a directory
grep: /etc/cron.monthly: Is a directory
/etc/cron.system_check:MAILTO=""
/etc/crontab:MAILTO=""
grep: /etc/cron.weekly: Is a directory

After all of this, the alerting worked fine.