Keycloak SSL renew certbot certificate

Question

I have a keycloak (docker) SSL system working with a certificate created by certbot, but upon renewing the certificate, the keycloak instance still show the invalid out of date certificate. I have checked using openssl that that certificate I created is valid and is in the /etc/x509/https folder. permissions on the files are fine. I even tried the following but nothing forced it to pickup the new certificate

1. restarting keycloak
2. signing into keycloak docker instance and running /opt/jboss/tools/x509.sh - it says it regenerated a new set of files but the date stamp seemed to imply it was still the old .jks and .pk12
3. moved those file from /opt/jboss/keycloak/standalone/configuration/keystores into a new temporary folder and ran the x509.sh again and it created two new file. I restarted the docker instance - but again it still shows the old certificate dates

Anyone got any idea on why the old certificate is not being refreshed? I believe this is a keycloak question rather than certbot. Any help would be appreciated.

Answer 1

Note : Answer working for Keycloak 9 (2020).

The simplest solution is to drop the container but it's not always desirable. However, there is another way.

AFAIK, x509.sh is supposed to be run only once per container life. You can take a look at docker-entrypoint.sh on the repository to verify that x509.sh is just run on container initialization and never again. In previous versions of docker-entrypoint.sh, x509.sh was run at every startup but it was doing nothing despite the messages it was printing.

Current x509.sh version is achieving the following steps:

• generating a random password:

  local PASSWORD=$(openssl rand -base64 32 2>/dev/null)
  

• creating a PKCS12 keystore with openssl:

  openssl pkcs12 -export \
  -name "${NAME}" \
  -inkey "${X509_KEYSTORE_DIR}/${X509_KEY}" \
  -in "${X509_KEYSTORE_DIR}/${X509_CRT}" \
  -out "${KEYSTORES_STORAGE}/${PKCS12_KEYSTORE_FILE}" \
  -password pass:"${PASSWORD}" >& /dev/null
  

• creating a JKS keystore from the PKCS12 one with keytool:

  keytool -importkeystore -noprompt \
  -srcalias "${NAME}" -destalias "${NAME}" \
  -srckeystore "${KEYSTORES_STORAGE}/${PKCS12_KEYSTORE_FILE}" \
  -srcstoretype pkcs12 \
  -destkeystore "${KEYSTORES_STORAGE}/${JKS_KEYSTORE_FILE}" \
  -storepass "${PASSWORD}" -srcstorepass "${PASSWORD}" >& /dev/null
  

• configuring the JKS keystore for Keycloak:

  $JBOSS_HOME/bin/jboss-cli.sh --file=/opt/jboss/tools/cli/x509-keystore.cli >& /dev/null
  

If you modify x509.sh and remove all redirections to /dev/null, you should see something like this:

Creating HTTPS keystore via OpenShift's service serving x509 certificate secrets..
Importing keystore /opt/jboss/keycloak/standalone/configuration/keystores/https-keystore.pk12 to /opt/jboss/keycloak/standalone/configuration/keystores/https-keystore.jks...
keytool error: java.io.IOException: keystore password was incorrect
HTTPS keystore successfully created at: /opt/jboss/keycloak/standalone/configuration/keystores/https-keystore.jks
{
    "outcome" => "failed",
    "failure-description" => "WFLYCTL0212: Duplicate resource [
    (\"subsystem\" => \"elytron\"),
    (\"key-store\" => \"kcKeyStore\")
]",
    "rolled-back" => true
}
{
    "outcome" => "failed",
    "failure-description" => "WFLYCTL0212: Duplicate resource [
    (\"subsystem\" => \"elytron\"),
    (\"key-store\" => \"kcKeyStore\")
]",
    "rolled-back" => true
}

It failed to modify Keycloak configuration with jboss-cli.sh. If you just remove keystores, and run x509.sh, the new randomly generated password will be different from the one in Keycloak configuration file. Since x509-keystore.cli is trying to add parameters, not update them, password in keystores and password in configuration won't match.

Here is an alternative version of x509.sh for renewal only whose key points are exposed below:

• extract currently used password from Keycloak configuration:

  local PASSWORD=$(/opt/jboss/keycloak/bin/jboss-cli.sh --connect --output-json '/subsystem=elytron/key-store=kcKeyStore:read-attribute(name="credential-reference")' |sed -rn 's;.+"result" *: *\{"clear-text" *: *"([^"]+)".*;\1;p')
  

• extract JKS keystore path from Keycloak configuration:

  local JKS_KEYSTORE_PATH=$(/opt/jboss/keycloak/bin/jboss-cli.sh --connect --output-json '/subsystem=elytron/key-store=kcKeyStore:read-attribute(name="path")' |sed -rn 's;.+"result" *: *"([^"]+https[^"]+)".*;\1;p')
  

• assume that PKCS12 keystore just differs by its extension:

  local PKCS12_KEYSTORE_PATH=${JKS_KEYSTORE_PATH%.*}.pk12
  

• now that you know the password and the keystore paths, update the PKCS12 keystore:

  openssl pkcs12 -export \
  -name "${NAME}" \
  -inkey "${X509_KEYSTORE_DIR}/${X509_KEY}" \
  -in "${X509_KEYSTORE_DIR}/${X509_CRT}" \
  -out "${PKCS12_KEYSTORE_PATH}" \
  -password pass:"${PASSWORD}"
  

• finally update the JKS one:

  keytool -importkeystore -noprompt \
  -srcalias "${NAME}" -destalias "${NAME}" \
  -srckeystore "${PKCS12_KEYSTORE_PATH}" \
  -srcstoretype pkcs12 \
  -destkeystore "${JKS_KEYSTORE_PATH}" \
  -storepass "${PASSWORD}" -srcstorepass "${PASSWORD}"
  

Complete script:

  function check_var() {
    local name=$1
    local value=$2
  
    if [ -z "$value" ]; then
      echo "$name is not defined."
      exit 1
    fi
  }
  
  function autoregenerate_keystore() {
    # Keystore infix notation as used in templates to keystore name mapping
    declare -A KEYSTORES=( ["https"]="HTTPS" )
  
    local KEYSTORE_TYPE=$1
    check_var "KEYSTORE_TYPE" $KEYSTORE_TYPE
  
    # reading password from configuration
    local PASSWORD=$(/opt/jboss/keycloak/bin/jboss-cli.sh --connect --output-json '/subsystem=elytron/key-store=kcKeyStore:read-attribute(name="credential-reference")' |sed -rn 's;.+"result" *: *\{"clear-text" *: *"([^"]+)".*;\1;p')
    check_var "PASSWORD" $PASSWORD
  
    # reading jks keystore path from configuration
    local JKS_KEYSTORE_PATH=$(/opt/jboss/keycloak/bin/jboss-cli.sh --connect --output-json '/subsystem=elytron/key-store=kcKeyStore:read-attribute(name="path")' |sed -rn 's;.+"result" *: *"([^"]+'$KEYSTORE_TYPE'[^"]+)".*;\1;p')
    check_var "JKS_KEYSTORE_PATH" $JKS_KEYSTORE_PATH
  
    if [ ! -f "${JKS_KEYSTORE_PATH}" ]; then
      echo "JKS keystore file does not exist!"
      exit 1
    fi
  
    # supposing that keystores were generated by x509.sh, hence pk12 keystore is in the same location.
    local PKCS12_KEYSTORE_PATH=${JKS_KEYSTORE_PATH%.*}.pk12
  
    if [ ! -f "${PKCS12_KEYSTORE_PATH}" ]; then
      echo "PKCS12 keystore file does not exist!"
      exit 1
    fi
  
    local X509_KEYSTORE_DIR="/etc/x509/${KEYSTORE_TYPE}"
    local X509_CRT="tls.crt"
    local X509_KEY="tls.key"
  
    local NAME="keycloak-${KEYSTORE_TYPE}-key"
  
    if [ ! -f "${X509_KEYSTORE_DIR}/${X509_KEY}" ] || [ ! -f "${X509_KEYSTORE_DIR}/${X509_CRT}" ]; then
      echo "X509 files does not exist!"
      exit 1
    fi
  
    echo "Renewing ${KEYSTORES[$KEYSTORE_TYPE]} keystore via OpenShift's service serving x509 certificate secrets.."
  
    openssl pkcs12 -export \
    -name "${NAME}" \
    -inkey "${X509_KEYSTORE_DIR}/${X509_KEY}" \
    -in "${X509_KEYSTORE_DIR}/${X509_CRT}" \
    -out "${PKCS12_KEYSTORE_PATH}" \
    -password pass:"${PASSWORD}"
  
    keytool -importkeystore -noprompt \
    -srcalias "${NAME}" -destalias "${NAME}" \
    -srckeystore "${PKCS12_KEYSTORE_PATH}" \
    -srcstoretype pkcs12 \
    -destkeystore "${JKS_KEYSTORE_PATH}" \
    -storepass "${PASSWORD}" -srcstorepass "${PASSWORD}"
  }
  
  autoregenerate_keystore "https" 

Name it x509-renewal.sh for example and copy it in your container:

$ docker cp x509-renewal.sh my-keycloak-container:/opt/jboss/tools/

then run it:

$ docker exec my-keycloak-container /opt/jboss/tools/x509-renewal.sh

Renewing HTTPS keystore via OpenShift's service serving x509 certificate secrets..
Importing keystore /opt/jboss/keycloak/standalone/configuration/keystores/https-keystore.pk12 to /opt/jboss/keycloak/standalone/configuration/keystores/https-keystore.jks...
Warning: Overwriting existing alias keycloak-https-key in destination keystore