OpenSSL s_client can't connect to Tomcat 7 via APR

Question

I'm currently in the process of upgrading my Tomcat servers to Tomcat 7 using the APR connector with SSLv3 disabled. Here is my connector:

<Connector port="8443" maxHttpHeaderSize="8192"
           maxThreads="150"
           enableLookups="false" disableUploadTimeout="true"
           acceptCount="100" scheme="https" secure="true"
           SSLEnabled="true" 
           SSLProtocol="TLSv1"
           SSLCertificateFile="${CP_ROOT}/security/tomcat.crt"
           SSLCertificateKeyFile="${CP_ROOT}/security/tomcat.key" />

Everything seems to be working properly... e.g. going to a page via HTTPS serves up that page correctly. However, we're using an F5 load balancer and as soon as I disabled SSLv3, the configured health monitor started failing for that node/port. After some troubleshooting on the F5 side, I decided to try to diagnose with OpenSSL:

$ openssl s_client -connect casrept2.tc.columbia.edu:8443/cas/monitor.jsp      
CONNECTED(00000003)
write:errno=54

Doing the same, but forcing TLSv1 (-tls1), I'm able to connect properly:

$ openssl s_client -connect casrept2.tc.columbia.edu:8443/cas/monitor.jsp -tls1
CONNECTED(00000003)
... cert chain, etc, etc

I'm wondering if that's what's causing the health monitor to fail. Either way though, I'm curious why I need to specifically force -tls1 for this to work. I would assume it should auto-negotiate the correct protocol?

Answer 1

What version openssl are you using for s_client? If it is 0.9.8 series, that defaults to protocols ssl2,ssl3,tls1(.0) which requires it to use SSL2 format ClientHello (but with contents which can negotiate up to tls1.0). openssl server in Tomcat/APR with tls1-only specified (instead of the default auto-detect) will either negotiate or give an alert back as appropriate for SSL3+ format, but for SSL2 format it just closes the TCP connection -- and at least on Windows it does so "abnormally" with RST which causes s_client to get a syscall error and display the message you see. (Although 54 is an errno value I don't recall seeing for RST; can you say what OS?)

By specifying s_client -tls1 you cause it to use SSL3+ format containing version 3.1 = TLS1.0 and it works. If you instead use -ssl3 it can't negotiate but it does get the somewhat more specific alert handshake failure instead of just a TCP error. If you use -no_ssl2 that uses SSL3+ format and negotiates TLS1.0, although that's less convenient here than -tls1.

If you use openssl 1.0.0 or 1.0.1 series for s_client they should default to minimum-SSL3 and (thus) SSL3+ hello format and work, unless something unusual was done in the build process.

I don't know what an F5 monitor does here -- or can do -- but it wouldn't surprise me if it tries SSL2 format hello on the theory that is -- or in the past was -- maximally interoperable. If so that would cause it to get a connection failure which it would understandably interpret as a problem with the server. You could put a trace such as www.wireshark.org or similar on (or near) your server and look to see exactly what F5 sends, and confirm your server responds with RST.