How Proxy SSL works on BIG-IP

1. Lab Scenario

Lab test results:

• Client completes 3-way handshake with BIG-IP and BIG-IP immediately opens and completes 3-way handshake with back-end server
• Upon receiving Client Hello on client-side, BIG-IP immediately sends Client Hello on server-side as it is
• BIG-IP copies same cipher suite list seen on client-side Client Hello to server-side Client Hello
• BIG-IP ignores what is configured on both Client SSL or Server SSL profiles.
• As soon as BIG-IP receives Server Hello it confirms two things:
• Is RSA the chosen key exchange mechanism in Cipher Suite on Server Hello message sent from back-end server?
• Does Certificate sent by back-end matches the one configured on Server SSL profile on BIG-IP?
• ONLY if both conditions above match BIG-IP proceeds with handshake. 
• Otherwise, BIG-IP sends a Handshake Failure (Fatal Alert) to both Server and Client with reset cause of illegal_parameter
• BIG-IP copies same Server SSL/Back-end Server certificate to Certificate message sent to Client on client-side
• BIG-IP completely ignores certificate you configured on Client SSL. It always uses the same server-side certificate.
• Assuming TLS handshake completes successfully BIG-IP is able to decrypt all client-side as well as server-side data which is the whole purpose of Proxy SSL.

2. How Proxy SSL works 

When Proxy SSL is enabled, BIG-IP does its best to match client-side to server-side connection in terms of negotiation and traffic to make it as transparent as possible to both client and back-end server and at the same allowing BIG-IP to decrypt traffic.

About data transparency

BIG-IP achieves transparency by copying whatever client and server sends back and forth.

About data decryption

BIG-IP has an extra configuration requirement for Proxy SSL configuration (according to K13385) that you should add the same certificate/key present on the back-end server to Certificate/Key fields on Server SSL proxy of BIG-IP. This way BIG-IP can decrypt both client and server sides of connection.

In practical terms and to achieve this, BIG-IP completely changes the original purpose of Server SSL Certificate and Key fields. Here's my config:

Certificate/Key fields above are no longer used for the purpose described in K14806, i.e. to independently authenticate BIG-IP to back-end server through Client Authentication. Instead, when Proxy SSL is enabled it is used to validate if Certificate sent by back-end server is the same one in this field. If so, BIG-IP also copies this certificate to Certificate message sent to Client on client-side.

Notice that when Proxy SSL is NOT bypassed Certificate configured on Client SSL profile is never used.

As soon as I sent first request to Wildcard forwarding VIP BIG-IP establishes TCP connection on client-side first and then immediately on server-side:

The next step is to forward exactly the same Client Hello we receive from Client on client-side to Server on server-side:

Now server sends Server Hello:

The first thing BIG-IP checks is key exchange mechanism as Proxy SSL has to use RSA (frame 12):

Now BIG-IP checks if Certificate inside Certificate message is the same as the one configured on Server SSL.

In this case I have used Certificate's unique serial number to confirm this.

On BIG-IP:

Now on Server's message (frame 12):

Now BIG-IP sends this same certificate in Server Hello message client-side and we can confirm from Serial number that it is the same:

From this moment handshake should complete successfully and BIG-IP is maintaining 2 separate connections using the same certificate/key pair on client and server side with the ability to decrypt both.

3. Troubleshooting Proxy SSL when BIG-IP is the culprit

Typically, if BIG-IP is the culprit it either because back-end server selected non-RSA key exchange cipher or because cert/key which are not supported.

In another test I used DHE on purpose and BIG-IP resets connection immediately after Server Hello message is received from back-end server which is typical sign of validation error:

I confirmed back-end server had selected ECDHE key exchange cipher which is not supported by Proxy SSL (frame 12):

In case you don't know yet here's how you work out key exchange cipher:

Disabling non-RSA cipher on back-end server did the trick to fix the above error as Proxy SSL only supports RSA key exchange.

4. Setting Up Proxy SSL on BIG-IP

I used very minimal configuration for this lab and the only thing I did was to create a wildcard forwarding virtual server using Standard VIP:

I enabled proxy-ssl on both Client and Server SSL profiles and added back-end server certificate (ltm3.crt) and key (ltm3.key) to Server SSL profile:

I also disabled (EC)DHE and explicitly configured RSA as key exchange mechanism in my back-end server.

I also confirmed back-end server was also using ltm3.crt/ltm3.key as it must match the one configured on Server SSL profile:

And it all worked fine:

Source:F5.com