Debugging Java SSL connection (OpenJDK-11 / FreeBSD-12.1)

1.8k views Asked by At

I am trying to debug why a javamail client cannot successfully connect to a mail server to forward messages. I have installed SSLPoke to see what is going on with the SSL connection as the Javamail client is embedded in a substantial web application (iDempiere).

This seems to be related to the java keystore used to establish the connection. But I cannot tell what is wrong with it.

The contents of the local keystore are:

JAVA_VERSION="11" keytool -list -v -keystore /opt/idempiere/idempiere-server/jettyhome/etc/keystore
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: accounting-2
Creation date: Dec 14, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: DC=ca, DC=harte-lyne, C=CA, ST=Ontario, L=Hamilton, O=Harte & Lyne Limited, OU=Networked Data Services, CN=accounting.harte-lyne.ca
Issuer: DC=ca, DC=harte-lyne, C=CA, ST=Ontario, L=Hamilton, O=Harte & Lyne Limited, OU=Networked Data Services, CN=CA_HLL_ISSUER_2016
Serial number: 20160054
Valid from: Fri Jul 31 20:00:00 EDT 2020 until: Sun Aug 31 19:59:59 EDT 2025
Certificate fingerprints:
     SHA1: 20:C4:82:9B:55:08:6C:6B:6D:C3:85:7C:52:5A:87:27:11:48:E9:B6
     SHA256: 98:26:68:02:9D:80:BD:34:B6:FD:93:A1:77:90:C1:5F:1D:75:1C:A7:1D:1B:BF:17:D6:B0:D7:83:78:2E:4E:23
Signature algorithm name: SHA512withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

Extensions: 

#1: ObjectId: 2.16.840.1.113730.1.4 Criticality=false
0000: 16 35 68 74 74 70 3A 2F   2F 63 61 2E 68 61 72 74  .5http://ca.hart
0010: 65 2D 6C 79 6E 65 2E 63   61 2F 43 41 5F 48 4C 4C  e-lyne.ca/CA_HLL
0020: 5F 49 53 53 55 45 52 5F   32 30 31 36 2F 63 72 6C  _ISSUER_2016/crl
0030: 2D 76 31 2E 63 72 6C                               -v1.crl


#2: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: caIssuers
   accessLocation: URIName: http://ca.harte-lyne.ca/CA_HLL_ISSUER_2016/ca.crt
]
]

#3: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: FD C6 20 77 C5 AA E8 34   43 99 C4 3D 5B 65 9A 3C  .. w...4C..=[e.<
0010: 2D 14 8E AF                                        -...
]
[L=Hamilton, DC=ca, DC=harte-lyne, C=CA, OU=Networked Data Services, O=Harte & Lyne Limited, ST=Ontario, CN=CA_HLL_ROOT_2016]
SerialNumber: [    02]
]

#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://ca.harte-lyne.ca/CA_HLL_ISSUER_2016/crl-v2.crl]
]]

#5: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [1.3.6.1.4.1.44880.100.10.10.3.1]
[PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.1
  qualifier: 0000: 16 1B 68 74 74 70 3A 2F   2F 63 61 2E 68 61 72 74  ..http://ca.hart
0010: 65 2D 6C 79 6E 65 2E 63   61 2F 43 50 53           e-lyne.ca/CPS

], PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.2
  qualifier: 0000: 30 34 1A 32 4C 69 6D 69   74 65 64 20 4C 69 61 62  04.2Limited Liab
0010: 69 6C 69 74 79 2C 20 73   65 65 20 68 74 74 70 3A  ility, see http:
0020: 2F 2F 63 61 2E 68 61 72   74 65 2D 6C 79 6E 65 2E  //ca.harte-lyne.
0030: 63 61 2F 43 50 53                                  ca/CPS

]]  ]
]

#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
  emailProtection
]

#7: ObjectId: 2.5.29.18 Criticality=false
IssuerAlternativeName [
  RFC822Name: [email protected]
  URIName: http://ca.harte-lyne.ca
]

#8: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  DigitalSignature
  Non_repudiation
  Key_Encipherment
]

#9: ObjectId: 2.16.840.1.113730.1.1 Criticality=false
NetscapeCertType [
   SSL client
   SSL server
   S/MIME
]

#10: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  RFC822Name: [email protected]
  DNSName: accounting.harte-lyne.ca
  DNSName: accounting
  DNSName: accounting.internal
  DNSName: accounting.harte-lyne.ca
  DNSName: accounting.internal.harte-lyne.ca
  DNSName: accounting-1
  DNSName: accounting-1.internal
  DNSName: accounting-1.harte-lyne.ca
  DNSName: accounting-1.internal.harte-lyne.ca
  DNSName: accounting-2
  DNSName: accounting-2.internal
  DNSName: accounting-2.harte-lyne.ca
  DNSName: accounting-2.internal.harte-lyne.ca
  DNSName: ledgersmb
  DNSName: ledgersmb.internal
  DNSName: ledgersmb.harte-lyne.ca
  DNSName: ledgersmb.internal.harte-lyne.ca
  DNSName: localhost
  DNSName: localhost.harte-lyne.ca
  IPAddress: 216.185.71.87
  IPAddress: 192.168.216.87
  IPAddress: 192.168.216.88
  IPAddress: 216.185.71.87
  IPAddress: 216.185.71.88
  IPAddress: 127.0.87.1
  IPAddress: 127.0.88.1
  IPAddress: 127.0.0.1
]

#11: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 23 12 19 8F 6E CB 1D 21   C2 7F 59 03 C6 69 B6 FB  #...n..!..Y..i..
0010: 41 99 B5 89                                        A...
]
]



*******************************************
*******************************************

This was created from a pkcs12 file containing:

openssl pkcs12 -in accounting-2.p12 | grep subject
Enter Import Password:
subject=CN = accounting.harte-lyne.ca, OU = Networked Data Services, O = Harte & Lyne Limited, L = Hamilton, ST = Ontario, C = CA, DC = harte-lyne, DC = ca
subject=CN = CA_ISSUER_2016, OU = Networked Data Services, O = Harte & Lyne Limited, L = Hamilton, ST = Ontario, C = CA, DC = harte-lyne, DC = ca
subject=CN = CA_HLL_ROOT_2016, ST = Ontario, O = Harte & Lyne Limited, OU = Networked Data Services, C = CA, DC = harte-lyne, DC = ca, L = Hamilton
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

Using:

JAVA_VERSION="11" keytool -importkeystore  -srckeystore accounting-2.p12  -destkeystore accounting-2.keystore -srcstoretype pkcs12 -alias accounting-2

cp -p accounting-2.keystore /opt/idempiere/idempiere-server/jettyhome/etc/keystore

The service I am connecting to uses a certificate issued by CA_ISSUER_2016 and refers to a ca_bundle which contains that issuer and its root certificates.

However, when I connect to this server using SSLpoke to test the validity of the keystore I get this:

JAVA_VERSION="11" java -Djavax.net.debug=ssl -Djavax.net.ssl.trustStore=/opt/idempiere/idempiere-server/jettyhome/etc/keystore  SSLPoke 192.168.216.32 465
javax.net.ssl|DEBUG|01|main|2020-12-18 09:42:33.441 EST|SSLCipher.java:438|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|DEBUG|01|main|2020-12-18 09:42:33.607 EST|SSLCipher.java:1840|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|01|main|2020-12-18 09:42:33.611 EST|SSLCipher.java:1994|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|WARNING|01|main|2020-12-18 09:42:33.660 EST|SSLSocketImpl.java:1550|handling exception (
"throwable" : {
  java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

It turns out that the java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty error arises from the fact that the keystore is password protected and the password was not provided.

After providing the password then the error obtained changes:

export PW=keystore_password

JAVA_VERSION="11" java -Djavax.net.debug=ssl -Djavax.net.ssl.trustStore=/opt/idempiere/idempiere-server/jettyhome/etc/keystore  -Djavax.net.ssl.trustStorePassword=$PW  -Djavax.net.ssl.keyStorePassword=$PW SSLPoke google.ca 443
JAVA_VERSION="11" java -Djavax.net.debug=ssl -Djavax.net.ssl.trustStore=/opt/idempiere/idempiere-server/jettyhome/etc/keystore  -Djavax.net.ssl.trustStorePassword=$PW  -Djavax.net.ssl.keyStorePassword=$PW SSLPoke google.ca 443javax.net.ssl|DEBUG|01|main|2020-12-18 10:13:45.561 EST|SSLCipher.java:438|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|ERROR|01|main|2020-12-18 10:13:45.904 EST|TransportContext.java:318|Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (
"throwable" : {
  sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I do not know how to interpret this. Does the message mean that the google certificate chain is missing from the keystore I am using? Or does it mean that the local keystore does not contain the certificates for the local host?

If I change the target to a service whose certificates are issued by the same issuer and root CA as the client I get the same error:

JAVA_VERSION="11" java -Djavax.net.debug=ssl -Djavax.net.ssl.trustStore=/opt/idempiere/idempiere-server/jettyhome/etc/keystore  -Djavax.net.ssl.trustStorePassword=$PW  -Djavax.net.ssl.keyStorePassword=$PW SSLPoke webmail.harte-lyne.ca 443
javax.net.ssl|DEBUG|01|main|2020-12-18 10:18:23.532 EST|SSLCipher.java:438|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|ERROR|01|main|2020-12-18 10:18:23.889 EST|TransportContext.java:318|Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (
"throwable" : {
  sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

What needs to be provided in the keystore for connections to work? I have no trouble connecting to our internal services using openssl s_client and our own certificates.

Further testing has changed things somewhat. I decided to add the intermediate and root certificates manually to a copy of the keystore. The error then changed:

JAVA_VERSION="11" keytool -import -trustcacerts -file /usr/local/etc/pki/tls/certs/CA_HLL_ISSUER_2016.crt -alias 'hartelyneissuer2016 [hll]' -keystore /root/ca_certs_accounting-2.keystore Enter keystore password:
Owner: DC=ca, DC=harte-lyne, C=CA, ST=Ontario, L=Hamilton, O=Harte & Lyne Limited, OU=Networked Data Services, CN=CA_ISSUER_2016 . . . Trust this certificate? [no]: yes Certificate was added to keystore

AVA_VERSION="11" keytool -import -trustcacerts -file /usr/local/etc/pki/tls/certs/CA_HLL_ROOT_2016.crt -alias 'hartelyneroot2016 [hll]' -keystore /root/ca_certs_accounting-2.keystore Enter keystore password:
Owner: L=Hamilton, DC=ca, DC=harte-lyne, C=CA, OU=Networked Data Services, O=Harte & Lyne Limited, ST=Ontario, CN=CA_HLL_ROOT_2016 . . . Trust this certificate? [no]: yes Certificate was added to keystore

Now SSLPoke gives this error:

JAVA_VERSION="11" java -Djavax.net.debug=ssl -Djavax.net.ssl.trustStore=/root/ca_certs_accounting-2.keystore  -Djavax.net.ssl.trustStorePassword=$PW  -Djavax.net.ssl.keyStorePassword=$PW SSLPoke webmail.harte-lyne.ca 443
javax.net.ssl|DEBUG|01|main|2020-12-18 10:33:08.306 EST|SSLCipher.java:438|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|ERROR|01|main|2020-12-18 10:33:08.626 EST|TransportContext.java:318|Fatal (CERTIFICATE_UNKNOWN): PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors (
"throwable" : {

According to SSL Exception on Java: Path does not chain with any of the trust anchors the previous error, unable to find valid certification path to requested target, is due to the fact that keytool only imports one certificate at a time, regardless of the length of the certificate chain provided in the pkcs12 file. So, adding the missing intermediate and root certificate removed that issue and resulted in the Path does not chain with any of the trust anchors error.

The problem now is: how does one establish a private CA root certificate as a trust anchor acceptable to java?

I had previously added the root and intermediate certificates to a copy of the cacerts file in JAVA_HOME:

keytool -keystore  /usr/local/openjdk11/lib/security/cacerts -storepass changeit -list | grep hll
hartelyneissuer2016 [hll], Dec 14, 2020, trustedCertEntry, 
hartelyneroot2016 [hll], Dec 14, 2020, trustedCertEntry, 

But this did not, and still does not, remove the Path does not chain with any of the trust anchors error. Why not?

0

There are 0 answers