I’m looking into migrating some k8s applications to KNative serving, this means I need to use Istio ingress gateway instead of Traefik ingress. However, I ran into issues trying to migrate basicAuth HTTP authentication so that the user must supply the correct username/password before request gets routed to the pod(this was done in Traefik using middleware annotations on the Ingress resource).
Ideally, I want to configure this authentication layer on a per-service basis, some of the services might not need authentication, some need to authenticate against differen secret resources. What’s a good approach to add HTTP authentication? Thanks!
I tried looking and found this doc that configures the gateway to talk with an external authentication service. And this doc that uses a sidecar container to perform auth before forwarding the request to the actual container.
I understand how they work, but I'm just wondering if there's a more light-weight approach to solving this particular issue I have.
I'm open to switching to a different authentication scheme, if they allow me to specify per-service secrets.
Update: I tried adding a per-service authentication EnvoyFilter to perform checks on the request header. However, this makes the KNative activator probe fail:
{"severity":"ERROR","timestamp":"2023-11-02T06:44:48.95474761Z","logger":"activator","caller":"net/revision_backends.go:398","message":"Failed to probe clusterIP 172.20.137.61:80","commit":"f60eb32","knative.dev/controller":"activator","knative.dev/pod":"activator-575d6749db-r9mdm","knative.dev/key":"kn-test/echo-server-00001","error":"unexpected status code: want [200], got 401","stacktrace":"knative.dev/serving/pkg/activator/net.(*revisionWatcher).checkDests\n\tknative.dev/serving/pkg/activator/net/revision_backends.go:398\nknative.dev/serving/pkg/activator/net.(*revisionWatcher).run\n\tknative.dev/serving/pkg/activator/net/revision_backends.go:443"}
So as of right now, I don't think the authentication check should be within the "boundary" of KNative. Perhaps at the entrypoint into our service mesh(istio ingress gateway)? Will look further.
After exploring many options, I eventually settled on adding the basic auth layer outside of Knative.
I stopped exposing the Knative services externally by removing the domain name from KnativeServing CR, so they can only be reached from within the cluster with "svc.cluster.local" url.
Then, I created Ingress resource for Nginx ingress controller to talk to the internal KNative cluster local url. I can easily configure the basic auth by setting Ingress annotations.
The tricky part is that 1. I need to have the Ingress point to the ExternalName service that's created as part of Knative service. 2. I need to manipute the "upstream-vhost" header to set the host to be the cluster local url.
But once it's setup, this provides the behaviour I want. If the user does not pass the authentication, then the request terminates on Nginx and Knative does not scale up.