Expose Kube API server via ingress running on the same cluster. SNI

Question

I have a K3S cluster with Traefik ingress. KubeAPI server is exposed on Port 6443 by default. I would like this to be hosted on port 443 - same port on which ingress runs, so via the ingress.

I tried creating a Ingress which uses the Kubernetes API service as a backend. But I get a Internal Server Error

I have already made the config to trust insecure backend certificate. I suspect the problem is occurring because API server uses Client certificates Traefik does not have the client certificate. There seems to be a way to configure a fixed client certificate in Traefik but that would be a security hazard as anyone can then connect to KubeAPI via Traefik. I would like the certificate to be passed through somehow.

Using Traefik as Layer 4 proxy would work but how do I do it on the same 443 port where other domains are hosted using Layer 7 proxying.

Ordinarily you can't have L4 and L7 proxy on the same port. But there are things like SNI which could provide a hint to Traefik about the domain name being used based on which Traefik can would know which backed to connect to and how - L4 or L7. So theoretically this should be possible. But I don't know whether Traefik supports this, and if it does, how to configure it in Kubernetes.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubeapi
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    ingress.kubernetes.io/auth-tls-insecure: "true"
middleware@kubernetescrd
spec:
  rules:
  - host: "kubeapi.foo.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: kubernetes
            port:
              name: https
  tls: # < placing a host in the TLS config will determine what ends up in the cert's subjectAltNames
  - hosts:
    - "*.foo.com"
    secretName: tls-secret # 

Answer 1

To expose the Kube API server via Traefik ingress on the same cluster while making sure the client certificate is passed through, you will indeed use the SNI (Server Name Indication) to differentiate traffic intended for the API server.

You have a first example with "Server Name Indication (SNI)", from Frank Rosner, but in your case, you would need the CRD implementation of a Traefik TCP router: a IngressRouteTCP, as seen in "Deploying a Kubernetes Cluster (AWS EKS) & an API Gateway secured by mTLS, with Terraform, External-DNS & Traefik - Part 2", form Aurélie Vache.

So make sure you have enabled the IngressRouteTCP CRD in your Traefik configuration, which allows TCP routing.
Then, you can define an IngressRouteTCP resource that specifies the entryPoint (port 443 in this case) and the SNI option.

Assuming that Traefik's TLS passthrough and SNI features are enabled:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: kubeapi-tcp
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
  - match: HostSNI(`kubeapi.foo.com`)
    services:
    - name: kubernetes
      port: 6443
  tls:
    passthrough: true

The entryPoints: websecure specifies that this IngressRouteTCP is for the HTTPS entry point. match: HostSNI('kubeapi.foo.com') uses SNI to route traffic to the API server. And tls: passthrough: true makes sure the TLS termination does not happen in Traefik, and the client certificate will be passed to the backend.

     Internet
        |
        ↓
  ┌──────────────┐
  │ Traefik      │
  │ Ingress      │
  └──────┬───────┘
         │
   TLS Passthrough
   (SNI for kubeapi.foo.com)
         │
         ↓
  ┌──────────────┐
  │  Kube API    │
  │  Server      │
  └──────────────┘

To apply the new IngressRouteTCP:

kubectl apply -f kubeapi-tcp-ingress.yaml

After applying that configuration, traffic to kubeapi.foo.com on port 443 should be routed to the Kube API server at port 6443 without TLS termination at Traefik, allowing the client certificate to be passed through.

That should allow you to run other services on port 443 with their own domain names and TLS certificates while still routing API server traffic correctly based on the SNI field, all through the same entry point managed by Traefik.