Communication issue between Spring Boot application (JHipster) and Keycloak through HTTPS

Question

In my team, we're trying to deploy a microservice stack based on JHipster (6.8.0) on OpenShift (4.2).

We have currently an issue when the gateway starts and tries to communicate with Keycloak through HTTPS (using Red Hat Single Sign On 7.3 based on Keycloak to be precise).

Here is the exception that is raised:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

We think it is because our gateway doesn't trust certificates from Keycloak. Indeed, this one is using organization certificates. We have logged in to the Keycloak administration interface on the realm the gateway is trying to connect with. And we have extracted certificates as X.509 binary-encoded DER files; thanks to Chrome browser functionality.

We first tried to simply add these certificates to the etc/ssl/certs/java/cacerts folder of our gateway container. To do so, we created these folders in the project jib repository, src/main/jib/etc/ssl/certs/java/cacerts, and copied the certificates to it.

We have generated our gateway Docker image using Maven and the jib:dockerBuild option. We pushed it to our Docker registry and deployed it to OpenShift. After a check in the OpenShift pod, certificates are well in place in etc/ssl/certs/java/cacerts. But we still get the same error as before.

We then tried to use a truststore. So we created one using this command for each certificate:

keytool -import -file path/to/certificate.cer -alias certificateAlias -keystore applicationTrustStore.jks

We checked that all certificates were properly added, thanks to this command:

keytool -list -v -keystore applicationTrustStore.jks

Then we added this applicationTrustStore.jks file into the src/main/jib/etc/ssl/certs/java/cacerts folder of our project and added this in the pom.xml:

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>${jib-maven-plugin.version}</version>
    <configuration>
        <from>
            <image>adoptopenjdk:11-jre-hotspot</image>
        </from>
        <to>
            <image>application:latest</image>
        </to>
        <container>
            …
            <jvmFlags>
                <jvmFlag>-Djavax.net.ssl.trustStore=etc/ssl/certs/java/cacerts/applicationTrustStore.jks</jvmFlag>
                <jvmFlag>-Djavax.net.ssl.trustStoreType=jks</jvmFlag>
                <jvmFlag>-Djavax.net.ssl.trustStorePassword=password</jvmFlag>
            </jvmFlags>
        </container>
        …
    </configuration>
</plugin>

Once again, we have generated and redeployed to OpenShift with no luck; still exactly the same issue.

We're certainly missing something obvious but we can't put our finger on it.

Answer 1

TL;DR — Try adding the certificate to the cacerts keystore of the JRE on which your application's Docker image is based.

---

The long-winded answer

Ahh yes! The old …PKIX path building failed… problem. Been there. And OpenShift was also involved to boot, in one instance.

Having come across that same error maybe a dozen or so times at this point, I would bet that the reason why your efforts so far had no effect, is because that javax.net.ssl.SSLHandshakeException is coming from the JRE running your application; not from whatever it is at /etc/ssl/certs/java/.

The solution

1. From your browser, obtain the Root CA certificate in question (what you referred to as: „…the Realm the gateway is trying to connect with…“)

   • Extract it the way you said you did in your question („…as X-509 binary encoded DER files…“)

2. Add the certificate to the cacerts keystore of the JRE on which your application's Docker image is based (<image>adoptopenjdk:11-jre-hotspot</image>)

   • An example Dockerfile to give you an idea of what you'll need to do:¹

FROM adoptopenjdk:11-jre-hotspot

COPY  stackexchange.cer /tmp/cert/certificate.cer

RUN  keytool -noprompt -import -alias deduper.answer -storepass changeit -keystore /opt/java/openjdk/lib/security/cacerts -file /tmp/cert/certificate.cer

CMD ["keytool", "-list", "-keystore",  "/opt/java/openjdk/lib/security/cacerts", "-alias", "deduper.answer", "-storepass", "changeit" ]

You can pull from Docker Hub, the image that Dockerfile builds…

$ docker pull deduper/ajrarn.soq.pkix.fix

Run it to observe that the cert with the alias was added…²

$ docker run -it deduper/ajrarn.soq.pkix.fix

Warning: use -cacerts option to access cacerts keystore
deduper.answer, Oct 10, 2020, trustedCertEntry,
Certificate fingerprint (SHA-256): E5:81:5A:DF:11:A9:0C:CC:51:8F:6A:99:D2:6C:67:16:29:D6:68:E1:EA:C2:C0:A7:E7:9B:84:09:AF:9C:29:14

If even that doesn't solve your problem, then the next thing I would suggest is to change this…

…
<jvmFlag>-Djavax.net.ssl.trustStore=etc/ssl/certs/java/cacerts/applicationTrustStore.jks</jvmFlag>
…

To this…

…
<jvmFlag>-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts/applicationTrustStore.jks</jvmFlag>
…

In case you didn't spot the difference, you've left out the leading forward slash from: /etc/…

---

^{1 You'll have to adapt however you build your image to jibe with the instructions in the example Dockerfile. 
2 I used the Stack Overflow cert for this experiment.} 

Answer 2

Thanks to the help of @deduper and @Chanseok Oh, I was able to solve the issue.

Few explanations in case some JHipster/Openshift users step around here.

First, I have tried to use keytool command to import organization certificate in /opt/java/openjdk/lib/security/cacerts. So I create tmp/cert subfolder in src/main/jib folder, put organization certificate inside and update src/main/jib/entrypoint.sh file like this:

#!/bin/sh

echo "The application will start in ${JHIPSTER_SLEEP}s..." && sleep ${JHIPSTER_SLEEP}
exec keytool -noprompt -import -alias alias -storepass changeit -keystore /opt/java/openjdk/lib/security/cacerts -file /tmp/cert/organization.cer

exec java ${JAVA_OPTS} -noverify -XX:+AlwaysPreTouch -Djava.security.egd=file:/dev/./urandom -cp /app/resources/:/app/classes/:/app/libs/* "com.your.company.App"  "$@"

It seems to work locally but when starting that on Openshift, I was getting a Permission denied when keytool was called because by default Openshift doesn't use root user. So, I finally remove my modifications and come back to the original entrypoint.sh file.

The workaround I use was:

• to extract cacerts file from adoptopenjdk:11-jre-hotspot (using docker cp command, more info: Copying files from Docker container to host)
• add certificate inside cacerts using keytool command on my workstation
• create these subfolders in src/main/jib: opt/java/openjdk/lib/security
• copy cacerts file inside

When creating Docker image using jib, cacerts in opt/java/openjdk/lib/security contains your organizational certificate and when starting on Openshift, communication with Keycloak is ok.

Answer 3

The JRE in adoptopenjdk:11-jre-hotspot never reads /etc/ssl/certs/java/cacerts by default. Most JREs actually read <JRE>/lib/security/cacerts by default (unless you set -Djavax.net.ssl.trustStore).

It is just that on many Linux distros, often <JRE>/lib/security/cacerts is a symlink to /etc/ssl/certs/java/cacerts. For example,

# ls -l /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts
lrwxrwxrwx    1 root     root            27 Jan  1  1970 /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -> /etc/ssl/certs/java/cacerts

In such cases, placing cacerts at /etc/ssl/certs/java will work. However, in adoptopenjdk:11-jre-hotspot, <JRE>/lib/security/cacerts is not a symlink, as shown below:

# ls -l /opt/java/openjdk/lib/security/cacerts 
-rw-r--r-- 1 root root 101001 Jul 15 09:07 /opt/java/openjdk/lib/security/cacerts

As @deduper explained, I'd put the file into /opt/java/openjdk/lib/security/.

And if you want to set -Djavax.net.ssl.trustStore to specify a different location, I think the path should be an absolute path as @deduper pointed out.