portainer not reachable anymore on domain with caddy as reverse proxy

Question

As a reverse proxy I'm using lucaslorentz/caddy-docker-proxy deployed in a docker swarm, so I can configure caddy using labels:

version: '3.7'

services:
  caddy_server:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    ports:
      - 80:80
      - 443:443
    networks:
      - caddy_controller
      - caddy
    environment:
      - CADDY_DOCKER_MODE=server
      - CADDY_CONTROLLER_NETWORK=10.200.200.0/24
    volumes:
      - caddy_data:/data
    deploy:
      replicas: 2
      labels:
        caddy_controlled_server:
        caddy.email: [email protected]
      placement:
        constraints: [node.role == manager]

  caddy_controller:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    networks:
      - caddy_controller
      - caddy
    environment:
      - CADDY_DOCKER_MODE=controller
      - CADDY_CONTROLLER_NETWORK=10.200.200.0/24
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  caddy_data: {}

networks:
  caddy:
    driver: overlay
    external: true

And I am using portainer configured with labels:

version: '3.2'

services:
  agent:
    image: portainer/agent:2.19.3
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agent_network
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ce:2.19.3
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "9443:9443"
      - "9000:9000"
      - "8000:8000"
    volumes:
      - portainer_data:/data
    networks:
      - agent_network
      - caddy
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]
    labels:
      caddy: portainer.example.com
      caddy.reverse_proxy: "{{upstreams 9000}}"


networks:
  agent_network:
    driver: overlay
    attachable: true
  caddy:
    driver: overlay
    external: true
    attachable: true


volumes:
  portainer_data:

There was a new version of portainer so I did an update. However, that update was not working correctly so the advice is to rollback to version 2.19.0 (which I did) since that rollback apparently supports rolling back the database. After that I updated again to 2.19.3.

However, now I'm unable to connect to the portainer service using the domain name defined in the label as portainer.example.com. I get this caddy error:

{
  "level": "error",
  "ts": 1702637831.2043304,
  "logger": "http.log.error",
  "msg": "dial tcp :9000: connect: connection refused",
  "request": {
    "remote_ip": "10.0.0.2",
    "remote_port": "46248",
    "client_ip": "10.0.0.2",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "portainer.example.com",
    "uri": "/favicon.ico",
    "headers": {
      "Pragma": ["no-cache"],
      "Cache-Control": ["no-cache"],
      "Dnt": ["1"],
      "Sec-Fetch-Dest": ["image"],
      "Sec-Fetch-Site": ["same-origin"],
      "Sec-Gpc": ["1"],
      "Referer": ["https://portainer.example.com/"],
      "Sec-Fetch-Mode": ["no-cors"],
      "Te": ["trailers"],
      "User-Agent": [
        "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0"
      ],
      "Accept": ["image/avif,image/webp,*/*"],
      "Accept-Language": ["en-US,en;q=0.5"],
      "Accept-Encoding": ["gzip, deflate, br"]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "portainer.example.com"
    }
  },
  "duration": 0.000365627,
  "status": 502,
  "err_id": "jzvc9j71y",
  "err_trace": "reverseproxy.statusError (reverseproxy.go:1267)"
}

Since I only updated, rolled back and then updated again I didn't expect to have an error in caddy.

I can access portainer using the ports defined in the yaml file and I can also shell into the caddy server containers (both of them) and perform a wget on portainer:9000 without problems.

Any idea what I can do so I can reach portainer again at portainer.example.com?