Sunday, 29 May 2022

F5 SSL Offloading

 In this post we’ll talk briefly about using your F5 Load Balancer to do SSL Offloading. Your F5 will be the termination point for all SSL Traffic so it can decrypt that traffic giving you more visibility into your incoming traffic. This is what we are after. From that point you can start using your WAF to make intelligent security decisions because now it can see the traffic.


Below are some benefits using your F5 for SSL Offloading:

Security Patches – If an SSL/TLS vulnerability comes out, you only need to apply the patches to the F5, not the Servers sitting behind the F5

Certificates – You only need to purchase certs for your F5 Virtual Servers and not the Servers sitting behind the F5

Routing Options – since you are decrypting traffic, you can now use things like the headers or URI to make routing decisions.

Improved Performance – because you are offloading the SSL/TLS processing (decryption) to the F5, your servers are now relieved of these CPU Intensive processes. So now your servers can do what they are meant to do, deliver content.

Now let’s look at a diagram.

F5 SSL Offloading

You will see users initiates an SSL/HTTPS connection from the outside. The F5 terminates this traffic, and offloads it or decrypts it. It now has full visibility into the traffic. But the regular web traffic is passed along to one of the servers in the pool unencrypted via HTTP.

1. Add SSL Certificates

The first thing you want to do is add the SSL Certificates to get the SSL termination setup. You’ll want to get these certs from whoever you purchase your public SSL certs like Network Solutions, GoDaddy, Verisign, LetsEncrypt, etc…

You will want to import both the .KEY and .CRT files here:

System > Certificate Management > Traffic Certificate Management > SSL Certificate List > Import…

F5 Cert 1

2. Build SSL CLIENT Profile

Here you will build out your Client SSL Profile which will get applied to the HTTPS Virtual Server.

Local Traffic > Profiles > SSL

F5 Cert 2

Important: make sure you click “Custom” to give you the “Certificate Key Chain” option.

F5 Cert 3

When you click “add“, you’ll get a prompt like the one below. Make sure you choose the .KEY and .CRT from step one.

F5 Cert 4

3. Virtual Server Setup

You’re now going to add the SSL Profile to the Virtual Server of your choice. Since we are working with “SSL Profile (Client)“, you do not put anything into “SSL Profile (Server)“.

Below shows clientssl but you want the profile you created above in step 2.

Local Traffic > Virtual Servers > Virtual Server List > {virtual server name}

F5 Cert 5

This is important. You should have two Virtual Servers, one for HTTP and one for HTTPS. We are working on the HTTPS profile. Make sure you choose your HTTP version for the “Default Pool“, not the HTTPS version which is what you would normally do.

This is where the HTTPS to HTTP traffic flow happens. You don’t want to do HTTPS back to the servers.

F5 Cert 6

4. Identify HTTPS Traffic

One issue you might run into is that the servers will be expecting HTTPS traffic and not HTTP Traffic. You’ll probably have your web servers setup to redirect HTTP>HTTPS. A way around this is to tell the servers that the traffic is legit and the F5 SSL Offloaded this traffic.

We will use X-Forwarded-Proto where SSL load balancer will be inserting the X-FORWARDED-PROTO header.  You will need to work with your dev team to get the web.config or PHP files configured for this.

To do this, you will want to create an iRule like this:

# Notify the backend servers that this traffic was SSL offloaded by the F5.

when HTTP_REQUEST {
  HTTP::header insert "X-Forwarded-Proto" "https";
}

Create your iRule here:

Local Traffic > iRules > iRules List

F5 Cert 7

Now you want to apply this iRule to your HTTPS Virtual Server:

Local Traffic > Virtual Servers > Virtual Server List > {virtual server name}

You’ll want to click on “Mange…” under “iRules”

F5 Cert 9

Like I mentioned above, you will need to get with your dev team to make code changes. Below are some examples for both web.confg and PHP:

WEB.CONFIG Example:
F5 Cert 11

PHP Example:

 $HTTPSEnabled = (isset($_SERVER['HTTP_X_FORWARDED_PROTO'])) ?
        strtolower($_SERVER['HTTP_X_FORWARDED_PROTO']) == "https":
        strtolower($_SERVER['HTTPS']) == "on";

F5 SMTP and SNMP Trap Alert Issues

 I ran across an F5 cluster that had issues with sending syslogs to our internal mail relay in order to get these logs via email. There were several issues I found.


First, test your SMTP and SNMP Traps config:
SMTP:

echo "SMTP Test Email" | mail -vs "Test email for KC" kcordero@cordero.me

SNMP Traps:

logger -p local0.notice "01070638:5: Pool /Common/pool_one member /Common/192.168.10.1:80 monitor status down."
logger -p local0.notice "01070727:5: Pool /Common/me member /Common/192.168.10.1:80 monitor status up."

logger -p local0.notice "01070640:5: Node 1.1.1.1 monitor status down."
logger -p local0.notice "01070728:5: Node 1.1.1.1 monitor status up."

For me both were not working so I had to get the SMTP working first because I was getting an error. Below is the error I was getting:

[admin@F5-bigip1:Active:In Sync] ~ # echo "SMTP Test Email" | mail -vs "Test email for KC" kcordero@cordero.me
send-mail: Cannot open localhost:25

Here’s the error in the log:

[admin@F5-bigip1:Active:In Sync] ~ # cat /var/log/maillog
Jan  4 09:52:45 F5-bigip1 err sSMTP[10482]: Unable to connect to "localhost" port 25.
Jan  4 09:52:45 F5-bigip1 err sSMTP[10482]: Cannot open localhost:25
Jan  4 12:24:01 F5-bigip1 err sSMTP[14036]: Unable to connect to "localhost" port 25.
Jan  4 12:24:01 F5-bigip1 err sSMTP[14036]: Cannot open localhost:25
Jan  4 12:33:40 F5-bigip1 err sSMTP[15968]: Unable to connect to "localhost" port 25.
Jan  4 12:33:40 F5-bigip1 err sSMTP[15968]: Cannot open localhost:25
Jan  4 12:33:50 F5-bigip1 err sSMTP[15993]: Unable to connect to "localhost" port 25.
Jan  4 12:33:50 F5-bigip1 err sSMTP[15993]: Cannot open localhost:25

So then I looked at the SMTP.CONF file:

[root@F5-bigip1:Active:In Sync] config # cat /etc/ssmtp/ssmtp.conf
#
# THIS IS AN AUTO-GENERATED FILE - DO NOT EDIT!!!
#
#
# /etc/ssmtp.conf -- a config file for sSMTP sendmail.
#
# See the ssmtp.conf(5) man page for a more verbose explanation of the
# available options.
#

# The person who gets all mail for userids < 500
# Make this empty to disable rewriting.
root=postmaster

# The place where the mail goes. The actual machine name is required
# no MX records are consulted. Commonly mailhosts are named mail.domain.com
# The example will fit if you are in domain.com and your mailhub is so named.
mailhub=mail.cordero.me:25

# Example for SMTP port number 2525
# mailhub=mail.your.domain:2525
# Example for SMTP port number 25 (Standard/RFC)
# mailhub=mail.your.domain
# Example for SSL encrypted connection
# mailhub=mail.your.domain:465

# Where will the mail seem to come from?
# RewriteDomain=

# The full hostname
# Hostname=

# Set this to never rewrite the "From:" line (unless not given) and to
# use that address in the "from line" of the envelope.
FromLineOverride=YES

# Use SSL/TLS to send secure messages to server.
#UseTLS=YES

# Use SSL/TLS certificate to authenticate against smtp host.
#UseTLSCert=YES

# Use this RSA certificate.
#TLSCert=/etc/pki/tls/private/ssmtp.pem

# Get enhanced (*really* enhanced) debugging information in the logs
# If you want to have debugging of the config file parsing, move this option
# to the top of the config file and uncomment

I noticed the "mailhub=" what commented out. At the same time, I removed the comment for "FromLineOverride=" because I want to override this.

After this, I was getting my test email alerts. So now onto the SNMP Traps. An important one for this is to make sure your OID & alert name matches the OID & name that's in "cat /etc/alertd/alert.conf" or else it won't work. Below is my "alert.conf" settings and a PIC of the "alert.conf" you need to match. Notice the OID & name in there. They need to match. The consultant who set this before me created them with different names. You can do that if their custom and your not using the default values.

My alert.conf config:

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.10";
    email toaddress="network@cordero.me"
    fromaddress="F5@cordero.me"
    body="Pool Member Down!"
}

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS_UP {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.11";
    email toaddress="network@cordero.me"
    fromaddress="F5@cordero.me"
    body="Pool Member Up!"
}

alert BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.12";
    email toaddress="network@cordero.me"
    fromaddress="F5@cordero.me"
    body="Node is Down!"
}

alert BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS_UP {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.13";
    email toaddress="network@cordero.me"
    fromaddress="F5@cordero.me"
    body="Node is Up!"
}

Screenshot of the "alert.conf" settings:
SNMP1

Another thing I had to do was to make sure MAILTO="" is at the top of the outputs below. Mine where NOT at the top and I had to move them. This was common if you upgraded your F5's software version.

crontab -e -u root

MAILTO=""
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
HOME=/

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

cat /etc/crontab

MAILTO=""
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
HOME=/

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

cat /etc/cron.d/0hourly

MAILTO=""
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
HOME=/
01 * * * * root run-parts /etc/cron.hourly

Below I just checked to see if the "MAILTO=" didn't have anything after the =.

grep -i mailto /etc/cron*

[admin@F5-bigip1:Active:In Sync] ~ # grep -i mailto /etc/cron*
grep: /etc/cron.d: Is a directory
grep: /etc/cron.daily: Is a directory
grep: /etc/cron.hourly: Is a directory
grep: /etc/cron.monthly: Is a directory
/etc/cron.system_check:MAILTO=""
/etc/crontab:MAILTO=""
grep: /etc/cron.weekly: Is a directory

After all of this, the alerting worked fine.

F5 Data Coming In but None Going Out

 99% of the time this is caused because the SNAT was not configured. It was over looked in the Virtual Server configuration.


Below you see that data is being received but not sent out. You see the “Bits” and “Packets” are 0:

To fix this, go to the “Source Address Translation” in the Virtual Server configuration and select “Auto Map“:

F5 Basic Tasks

 I’ll try to add to this post some of the F5 tuning settings you can configure for specific tasks.


One thing to keep in mind, a “session” is when you have multiple transactions between the same client and server’s established TCP connection. There will be multiple TCP connections through the same session in which case session persistence is required to maintain a session.

#POOL LEVEL – ACTION ON SERVICE DOWN
The first thing I want to talk about are the setting under the pools “Action on Service Down“. You have four options here: Reject, Drop, Reselect, & None.

Reject
The F5 will cut any active connection by sending an RST to both the server and client immediately AFTER a pool member transitions to a DOWN state. It will remove the connection from the LTM connection table. So, it basically closes both sides of the connection DOWN. Since it forces the client sides connection to close, it’s the quickest option to close sessions down.

Drop
The F5 will silently drop any new client packets sent on established connection. The connection remains up until:

1. The LTM idle timer related to the connection times out
2. either side closes the connection

Reselect
The F5 will choose another pool member if there’s one available and reestablish a connection. You want to use this when the client can continue with a new server. Just remember the request at the time of the switchover will be lost.

None (Default)
The F5 will keep sending data on established connections as long the client is sending, and the server is responding. Use this when you don’t want the F5 to intervene on either side of the connection.

#NODES LEVEL – DISABLE OR FORCE OFFLINE
Another setting I’d like to point out is a setting people use when they want to “remove” a node from a cluster to do maintenance or some other task. I quote remove because you’re not physically removing it from the cluster, but it will stop receiving traffic and not participate in the cluster.

These two options are in the Nodes tab and those are “Disable” and “Force Offline“. One note is that you want to do this at the node level because that node could be tied to different Pools. You’ll notice you can also do this at the Pool level. If you do it at the Pool level, you’ll have to do for each Pool. This is not efficient so it’s better to do it at the Node level.

So, what does each do?

Disabled:
Connections that are already established and everything in the persistence table will continue to process. New connections will be accepted only if the connection belongs to an existing persistence session.

Forced Offline:
F5 continues to manage connections already established only.
Only connection that already established will stayed connected until they time out. NO new connections are allowed.

In both cases, the F5 will remove the connections but the second will remove them quicker.

F5 – SSL Options (SSL Offloading vs SSL Bridging vs SSL Passthrough)

 #SSL OFFLOADING

Consider a conventional client-server architecture in which you have a client and a web or application server. HTTPS is used to establish the connection (with SSL encryption). You probably haven’t considered the number of resources these servers must consume to decrypt, process, and re-encrypt client requests before sending them back to the clients.

The BIG-IP system manages decryption and re-encryption using the SSL Offloading technique. The client delivers encrypted traffic to BIG-IP utilizing this manner. As a result, the BIG-IP decrypts the client communication and sends the decrypted traffic to the server. The BIG-IP encrypts the communication from the server to the client and sends it back to the client. This eliminates the additional overhead of encryption and decryption on the server. All server resources can now be fully utilized for other purposes, such as serving application content or performing other tasks for which they were designed.

To summarize, SSL offloading on load balancers like BIG-IP LTM allows a web server to be relieved of the processing effort of encrypting and decrypting data.

Key Takeaways:
-Security Patches – If an SSL/TLS vulnerability comes out, you only need to apply the patches to the F5, not the Servers sitting behind the F5
-Certificates – You only need to purchase certs for your F5 Virtual Servers and not the Servers sitting behind the F5
-Routing Options – since you are decrypting traffic, you can now use things like the headers or URI to make routing decisions.
-Improved Performance – because you are offloading the SSL/TLS processing (decryption) to the F5, your servers are now relieved of these CPU Intensive processes. So now your servers can do what they are meant to do, deliver content.


#SSL BRIDGING

SSL Re-Encryption or SSL Full Proxy are other terms for the same thing. The BIG-IP system accepts encrypted inbound traffic and decrypts it for traffic analysis purposes using this mechanism. However, it re-encrypts the connection before transmitting it to the destination servers. When the edge device performs a deep packet inspection to ensure that the contents of the SSL-encrypted transmission are safe, or when unencrypted traffic crossing the internal network poses a security risk, SSL bridging can be advantageous.

Key Takeaways:
-Each site has its own SSL session.
-Each segment’s communication is secure.
-Secure ports, such as port 443, are configured on servers so .
-BIG-IP has the capacity to read the content of traffic.
-All of the persistence settings are available.

ssl-bridging.png

#SSL PASSTHROUGH

SSL passthrough is one of the key configuration options for load balancing encrypted online traffic. Passing data through a load balancer to a server without decrypting it is known as SSL passthrough. The data is encrypted as it travels through the load balancer thanks to an SSL pass through. As a result, installing an SSL certificate on the load balancer is not required for proxy SSL passthrough settings. SSL certificates are installed on the backend server because the load balancer does not handle SSL connections.

In this case, it is the server that will perform the process of decrypting all incoming SSL traffic. Using SSL passthrough requires a higher processing unit on the part of the servers. This is why it is not recommended for larger deployments. It also restricts some capabilities of a load balancer. The SSL proxy pass-through does not inspect traffic or intercept SSL sessions on network devices before reaching the server, as it simply passes along encrypted data.

Let’s take a quick look at the configuration for this method on BIG-IP LTM. Because it’s only a pass-through, LTM can’t read the headers, which limits persistence. Only non-SSL information in the packet, such as the source and destination IP addresses, can be used to preserve persistence. Client SSL and Server SSL profiles should not be combined. Because you can’t use an HTTP profile to optimize layer seven traffic, you can’t use it. Cookie persistence isn’t an option.

This configuration is typically used when the apps being served cannot ingest encrypted traffic or when web application security is a concern.

ssl-passthrough.png



F5 Upgrade Hotfix and Point Release in a HA Pair

 Order of Operation:

1. Upgrade STANDBY
2. Then Upgrade PRIMARY

A point release or hotfix is going from 15.1.0 to 15.1.0.x.

1. Validating the Configuration

On F5-B (STANDBY):

tmsh
load /sys config verify

This command doesn’t load it but does a ‘test’ load. You are checking to see if there are any problems uploading after the upgrade.

2. Verify the Service Check Date

When looking at the images on the F5 site, look for “License Check Date” and make sure you have a later date than what’s posted on the site.

For example; if it’s 2019-11-05, it has to be a date past that, like 2019-12-01

q
grep "Service check date" bigip.license

If you are before the date, you will have to relicense the F5. Check out:
“K7727: License activation may be required before a software upgrade for the BIG-IP or Enterprise Manager system”
“K7752: Licensing the BIG-IP system”

3. Synchronizing the Config

Make sure the configuration is in synch before upgrading; if you are not in Sync, you need to sync it here:

On F5-A (ACTIVE):

Device Management > Overview

You want to sync with the device with the most recent changes.

4. Creating and Saving a UCS Archive (AKA BACKUP)

On F5-B (STANDBY):

System > Archives

Click on archive name, download, and save to a separate secure location

5. Importing the ISO File

System > Software Management > Image List > Import…

You will see it in the “Available images” list.

Check out:
“K167: Downloading software and firmware from F5”

6. Verifying the MD5 checksum

cd images/
ls
md5sum -c "filname.iso.md5"

Check out:
“K8337: Verifying the MD5 checksum for the downloaded F5 software file”

7. Disabling the “Automatic with Incremental Sync” Option (ON PRIMARY F5)

If your sync type is set to automatic, you should temporarily set it manual.

On F5-A (ACTIVE):

Device Groups > {Group Name} > Sync Type

Three Options:
1. Automatic with Incremental Sync
2. Manual with Incremental Sync
3. Manual with Full Sync

You want to change it from option 1 above to option 2 temporarily.

Select “Update” to commit Sync Type change

8. Installing and Rebooting to the New Version

On F5-B (STANDBY):

System > Software Management

Look at the “Boot Location” that’s both active and default. If it’s HD1.1, then create a new one HD1.2.

a. Check the box next to the new available image
b. Click “Install
c. “Select Disk” – you might have more than one, but it’s typically HD1 or MD1
d. “Volume set name:” type in 2
e. Click “Install

Verify status:

watch "tmsh show sys sof status"

ctrl-c to break out

f. Go to “Boot Locations
g. Click on new HD#
h. Keep “Install Configuration” to No – this is only if you made any config changes
i. Click “Activate

Verify Progress:
Use console connection

On F5-A (ACTIVE), you will see “Disconnected” for the Current ConfigSync State.

When you log back onto the F5-B, you will see its status is “Changes Pending.

IMPORTANT:
When they are on different versions, you do NOT want to synchronize!

9. Verifying the New Point Release Version is Active on the Newly Patched System

On F5-B (STANDBY):

System > Software Management

You should see that the new version is both Active and Default Boot.

Verify your Virtual Servers have loaded:

Local Traffic > Virtual Servers

10. Force Failover to Newly Patched System

After verifying everything loaded OK, it’s now ready to take Traffic.

On F5-A (ACTIVE):

Device Management > Traffic Groups

Check the box for the Traffic Group
Click “Force to Standby…
Click “Force to Standby

Look at the top left to the status; you should see F5-A is “STANDBY.

On F5-B (STANDBY):

You should see the F5-B is “ACTIVE,” but the with “Changes Pending” still.

Verify Traffic Flow:
Check to make sure Traffic is flowing properly

REPEAT PROCEDURES FOR A!

11. Repeat these Steps for F5-A

Validating the configuration
Verifying the Server check date
Creating and saving the UCS archive
Importing the ISO
Verifying the MD5 checksum
Installing and rebooting to the new version

12. Installing and Rebooting to the New Version & Verifying the New point Release Version is Active on the Newly Patched System

On F5-A (ACTIVE):

System > Software Management > Boot Locations

Click on new HD# and Verify Info
Click “Activate

Look for error messages and verify the Virtual Servers (pool members) are up.

13. Forcing a Failover Back to F5-A

On F5-B (STANDBY):

Device Management > Traffic Groups

Check the box for the Traffic Group
Click “Force to Standby…
Click “Force to Standby

On F5-A (ACTIVE):

You should see it’s “ACTIVE” and both are running the same point release.

Check both:

System > Software Management > Image List

Look to see if the new image is “Yes” for both Active and Default Boot.

14. Performing the Final ConfigSync

Now that both F5 are running the same code version, it’s time to Sync.

On F5-A (ACTIVE):

Device Management > Overview

Devices:
Recent Changes – should be on the F5-A, and it will show A because it’s the most recent loaded after the reboot and upgrade

BEST PRACTICE – DO NOT MAKE ANY CHANGES DURING AN UPGRADE

Make sure you select the F5-A
Make sure “Sync Options” is set to “Push the selected device configuration to the group”
Click “Sync

Everything should be back to normal and synced.

15. OPTIONAL: Restoring the “Automatic with Incremental Sync” Option

If this was set to automatic before this whole process, we could now restore it.

Device Groups > {Group Name} > Sync Type

Three Options:
1. Automatic with Incremental Sync
2. Manual with Incremental Sync
3. Manual with Full Sync

You want to change if from option 2 above back to option 1

Select “Update” to commit Sync Type change

iRule

  iRule: -- o iRule is a powerful and flexible feature within the BIG-IP local traffic management (LTM). o IRule is a powerful & flexibl...