Treafik uses DEFAULT CERT instead of using Let's Encrypt wildcard certificate

Question

I try to setup Traefik to get certificates from Let's Encrypt using DNS challenge and secure a whoami app with this certificate. I manage to get the certificate (well present in the acme.json file) but my IngressRoute doesn't use these certificate for the route.

My cluster is a K3D cluster. I deploy Traefik v2 from the official Helm Chart : helm install traefik traefik/traefik -f traefik-values.yaml

I defined these values for the chart :

additionalArguments:
  - --log.level=TRACE
  - --certificatesresolvers.le.acme.email=<MY_EMAIL>
  - --certificatesresolvers.le.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
  - --certificatesresolvers.le.acme.dnschallenge=true
  - --certificatesresolvers.le.acme.dnschallenge.provider=route53
  - --certificatesresolvers.le.acme.dnschallenge.delayBeforeCheck=60
  - --certificatesresolvers.le.acme.dnschallenge.resolvers=8.8.8.8:53
  - --certificatesresolvers.le.acme.storage=/data/acme.json
  - --entrypoints.web.http.redirections.entryPoint.to=:443
  - --entrypoints.web.http.redirections.entryPoint.scheme=https
persistence:
  enabled: true
  path: /data
env:
  - name: AWS_REGION
    value: eu-west-1
  - name: AWS_HOSTED_ZONE_ID
    value: <MY_AWS_HOSTED_ZONE_ID>
  - name: AWS_ACCESS_KEY_ID
    value: <MY_AWS_ACCESS_KEY_ID>
  - name: AWS_SECRET_ACCESS_KEY
    value: <MY_AWS_SECRET_ACCESS_KEY>

Deployment, Service and IngressRoute for whoami app :

kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoami
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami:v1.5.0
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  labels:
    app: whoami
spec:
  type: ClusterIP
  ports:
    - port: 80
      name: whoami
  selector:
    app: whoami
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: app-tls
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`test.mydomain.com`) || Path(`/whoami`)
      services:
        - name: whoami
          port: 80
  tls:
    certResolver: le
    domains:
      - main: "*.test.mydomain.com"

In the logs, I can see :

time="2020-09-24T14:04:04Z" level=debug msg="legolog: [INFO] acme: Registering account for MY_EMAIL"
time="2020-09-24T14:04:04Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] acme: Obtaining bundled SAN certificate"
time="2020-09-24T14:04:04Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/118300931"
time="2020-09-24T14:04:04Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] acme: use dns-01 solver"
time="2020-09-24T14:04:04Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] acme: Preparing to solve DNS-01"
time="2020-09-24T14:04:05Z" level=debug msg="legolog: [INFO] Wait for route53 [timeout: 2m0s, interval: 4s]"
time="2020-09-24T14:05:16Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] acme: Trying to solve DNS-01"
time="2020-09-24T14:05:16Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] acme: Checking DNS record propagation using [8.8.8.8:53]"
time="2020-09-24T14:05:20Z" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 4s]"
time="2020-09-24T14:06:24Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] The server validated our request"
time="2020-09-24T14:06:24Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] acme: Cleaning DNS-01 challenge"
time="2020-09-24T14:06:25Z" level=debug msg="legolog: [INFO] Wait for route53 [timeout: 2m0s, interval: 4s]"
time="2020-09-24T14:07:21Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] acme: Validations succeeded; requesting certificates"
time="2020-09-24T14:07:23Z" level=debug msg="legolog: [INFO] [*.test.mydomain.com] Server responded with a certificate."

And then :

time="2020-09-24T14:07:24Z" level=debug msg="Looking for provided certificate(s) to validate [\"*.test.mydomain.com\"]..." providerName=le.acme
time="2020-09-24T14:07:24Z" level=debug msg="No ACME certificate generation required for domains [\"*.test.mydomain.com\"]." providerName=le.acme

When I reach localhost/whoami from the browser, I can see the whoami app but the used certificate is the default cert from Traefik. The issue is the same with a non-wildcard certificate.

Why is the LE certificate not used for my route ?

Thank you in advance for your help.

Answer 1

I've just moved my website from new.example.com to example.com that was linked to the old version of the website hosted on the different server.

Instead of an automatic Let's encrypt certificate, traefik had used the default certificate. In one hour after the dns records was changed, it just started to use the automatic certificate. I haven't made an updates in configuration. I think it might be related to this and this issues posted on traefik's github.

If it is, in fact, related to the "chicken-and-egg problem as the domain shouldn't be moved to the new server before the keys work, and keys can't be requested before the domain works", I would recommend to use user-defined certificates for 24 hours after dns updates. Then it should be safe to fall back to automatic certificates.

Some details

I used the acme configuration from the docs:

certificatesResolvers:
  myresolver:
    acme:
      email: [email protected]
      storage: acme.json
      httpChallenge:
        # used during the challenge
        entryPoint: web

The weird thing was that /etc/traefik/acme/acme.json contained private key, though I don't know how it's supposed to work.

{
  "letsencrypt": {
    "Account": {
      "Email": "[email protected]",
      "Registration": {
        "body": {
          "status": "valid",
          "contact": [
            "mailto:[email protected]"
          ]
        },
        "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/*******"
      },
      "PrivateKey": "*******************************************",
      "KeyType": "4096"
    },
    "Certificates": null
  }
}

Also, I used docker and restarted container for couple of times without no lack. After the last restart it just started to work.

Answer 2

Ensure that the /etc/traefik/certs folder exists (you need to create it manually). To find/confirm the location, look for the production: acme: storage: entry in the certificateResolver section of traefik.yml