Problems connecting via HTTPS/SSL through own Java client

Question

I'm trying to establish a connection to trackobot.com to receive some JSON data. The server only allows connections through HTTPS/SSL. Here is the code:

java.lang.System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");
URL url = new URL("https://trackobot.com/profile/history.json?username=USER&token=TOCKEN");     
InputStream is = url.openStream(); 
JsonParser parser = Json.createParser(is);

openSteam throws javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

I read through several posts related to similar problems but none of the suggestions helped. The appropriate certificate is in my truststore. When I try to connect to, for example, google.com there is no error. So, the problem seems to be in the handshake-specifics of the server I'm trying to connect to.

I ran my code using -Djavax.net.debug=ssl returning this:

keyStore is : 
keyStore type is : jks
keyStore provider is : 
init keystore
init keymanager of type SunX509
trustStore is: /Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre/lib/security/cacerts
trustStore type is : jks
trustStore provider is : 
init truststore

[Here I removed hundreds of „adding as trusted cert“:… lines]

trigger seeding of SecureRandom
done seeding SecureRandom
Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256
Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
main, setSoTimeout(0) called
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 for TLSv1.1
%% No cached client session
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1433943269 bytes = { 109, 198, 189, 148, 62, 6, 19, 126, 179, 214, 250, 99, 207, 117, 162, 47, 62, 176, 222, 247, 98, 29, 155, 63, 255, 100, 171, 187 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA, MD5withRSA
Extension server_name, server_name: [type=host_name (0), value=trackobot.com]
***
main, WRITE: TLSv1.2 Handshake, length = 229
main, READ: TLSv1.2 Alert, length = 2
main, RECV TLSv1.2 ALERT:  fatal, handshake_failure
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1991)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1104)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1511)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1439)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at java.net.URL.openStream(URL.java:1038)
    at trackbot.readHistory(trackbot.java:37)
    at hsanalytics.main(hsanalytics.java:6)
Ende

Furthermore, I use openssl and a perl script (analyze-ssl.pl) to check the server.

For openssl s_client -connect trackobot.com:443 I got:

CONNECTED(00000003)
depth=2 /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/C=CH/CN=www.trackobot.com/emailAddress=df1c792ce8e2fc342c0c63c2fab9c6fe-1805689@contact.gandi.net
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
 1 s:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
 2 s:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGaTCCBVGgAwIBAgIHBaEI9iSK1jANBgkqhkiG9w0BAQsFADCBjDELMAkGA1UE
BhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBE
aWdpdGFsIENlcnRpZm
[...]
bnCZTkntRP7wPfw6DpPdJzt8BD0Rpp0B8fVUkqkUujP
FEgspzHXqvAp3gzDuNVlElZ4pxSC/06x9xlPua4KnnKIPMVK0DjyXKdPgUaw6YH9
I3SprrGd/B5AoxdPYDM1qRGC+hto3YDnAb29CRFx13mfiEF9En6YrmlZMwJ/dMjo
RcvkqpjoxTLODmX9gWgdJ27Ublq/4f/Q9nlVfx4v00eYyqyMYY6IMlOUWEBvWoAv
zorzLLY9PmepXJtkCw==
-----END CERTIFICATE-----
subject=/C=CH/CN=www.trackobot.com/emailAddress=df1c792ce8e2fc342c0c63c2fab9c6fe-1805689@contact.gandi.net
issuer=/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
---
No client certificate CA names sent
---
SSL handshake has read 5848 bytes and written 328 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: A868799D47C550929ADF026FDC48CABD2444C96FDDAB86036196029BF7754D1B
    Session-ID-ctx: 
    Master-Key: 6C0E428129970C6B1E358E134B12125373BED6FF50D55004A68A9042AD4E51C6D70BB8480266CC1BD1F11B093E212BFC
    Key-Arg   : None
    Start Time: 1433943895
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

For analyze-ssl trackobot.com:443 I got:

-- trackobot.com port 443
 ! server sent unused chain certificate '/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority'
 ! server sent unused chain certificate '/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority'
 * maximum SSL version  : TLSv1_2 (SSLv23)
 * supported SSL versions with handshake used and preferred cipher(s):
   * handshake protocols ciphers
   * SSLv23    TLSv1_2   ECDHE-RSA-AES256-GCM-SHA384
   * TLSv1_2   TLSv1_2   ECDHE-RSA-AES256-GCM-SHA384
   * TLSv1_1   TLSv1_1   ECDHE-RSA-AES256-SHA
   * TLSv1     TLSv1     ECDHE-RSA-AES256-SHA
   * SSLv3     FAILED: SSL connect attempt failed because of handshake problems error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure 
 * cipher order by      : server
 * SNI supported        : ok
 * certificate verified : ok
 * chain on 5.102.146.151
   * [0/0] bits=2048, ocsp_uri=http://ocsp.startssl.com/sub/class1/server/ca, /C=CH/CN=www.trackobot.com/emailAddress=df1c792ce8e2fc342c0c63c2fab9c6fe-1805689@contact.gandi.net SAN=DNS:www.trackobot.com,DNS:trackobot.com
   * [1/1] bits=2048, ocsp_uri=http://ocsp.startssl.com/ca, /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
   * [2/-] bits=4096, ocsp_uri=, /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
   * [-/2] bits=4096, ocsp_uri=, /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
 * OCSP stapling        : no stapled response
 * OCSP status          : good (soft error: no ocsp_uri for /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority)

Versions:

Mac OSX 10.10.3
OpenSSL 0.9.8zd 8 Jan 2015
java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)

Can anyone spot where my Java program fails? What can I do to meet the handshake requirements of the server? I this really the issue?

Answer 1

According to https://www.ssllabs.com, the server supports cipher suites

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA 
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA 

They are listed as "unavailable cipher suite" as you can see in the debug messages.

In the JRE/lib/security/local_policy.jar, we see

// Some countries have import limits on crypto strength. This policy file
// is worldwide importable.

grant {
    permission javax.crypto.CryptoPermission "DES", 64;
    permission javax.crypto.CryptoPermission "DESede", *;
    permission javax.crypto.CryptoPermission "RC2", 128, 
                                     "javax.crypto.spec.RC2ParameterSpec", 128;
    permission javax.crypto.CryptoPermission "RC4", 128;
    permission javax.crypto.CryptoPermission "RC5", 128, 
          "javax.crypto.spec.RC5ParameterSpec", *, 12, *;
    permission javax.crypto.CryptoPermission "RSA", *;
    permission javax.crypto.CryptoPermission *, 128;
};

Download and install "(JCE) Unlimited Strength Jurisdiction Policy Files" - http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html - and I can confirm that the problem is solved. The read me file says

Due to import control restrictions of some countries, the version of the JCE policy files that are bundled in the Java Runtime Environment, or JRE(TM), 8 environment allow "strong" but limited cryptography to be used. This download bundle (the one including this README file) provides "unlimited strength" policy files which contain no restrictions on cryptographic strengths.

Answer 2

Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, ...

There are no AES256 cipher suites offered by you Java client.

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA

Because they are not available in your application. I'm not an Java expert but either these are not available in your Java or they need to be explicitly enabled. They are needed because the server only supports AES256 ciphers:

 $ perl analyze-ssl.pl -v3 --all-ciphers trackobot.com
 ...
* supported ciphers with SSLv23 handshake
 * TLSv1_2 ECDHE-RSA-AES256-GCM-SHA384
 * TLSv1_2 ECDHE-RSA-AES256-SHA384
 * TLSv1_2 ECDHE-RSA-AES256-SHA
 * TLSv1_2 DHE-RSA-AES256-GCM-SHA384
 * TLSv1_2 DHE-RSA-AES256-SHA256
 * TLSv1_2 DHE-RSA-AES256-SHA

It might be that the version of Java you use has no support for AES256 because of export regulations, see https://knowledge.safe.com/articles/Error_Unexpected_Behavior/Enabling-AES256-in-the-Java-Runtime-Environment-for-Single-Sign-On

Answer 3

Thanks to Steffen Ullrich's tipp I checked the ciphers java had available. Apparently, in Java 8 you don't have unlimited strength for your ciphers. For example, in my case my program couldn't use a AES 256bit cipher which the server required.

To solve this, oracle provides a bundle of policy files that allow for unlimited strength encryption. You can find it here.

The README states:

Due to import control restrictions of some countries, the version of the JCE policy files that are bundled in the Java Runtime Environment, or JRE(TM), 8 environment allow "strong" but limited cryptography to be used.

Just download the bundle and replace appropriate files as stated in the install instructions. After I did that, the handshake worked like a charm.

Answer 4

You probably need upgrade your JDK, we had similar problem on our linux server. We tried different approaches. Nothing seemed to work including installing new JCE.

There is one bug in the JDK about SSL Connection HostnameVerifier that disables SNI extension causing a handshake failure: http://www.oracle.com/technetwork/java/javase/2col/8u141-bugfixes-3720387.html.

We upgraded to the latest jdk 8u162, everything looks good now.