I run the kserve example for sklearn-iris and got `302 Found`

Question

I run the kserve example for sklearn-iris and got `302 Found`

825 views Asked by TaeUk Noh At 25 November 2024 at 10:30

serving exam model

create namespace
```
$ kubectl create namespace kserve-test
```

create InferenceService

$ kubectl apply -n kserve-test -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
EOF

check

$ kubectl get inferenceservices sklearn-iris -n kserve-test

NAME           URL                                           READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION                    AGE
sklearn-iris   http://sklearn-iris.kserve-test.example.com   True           100                              sklearn-iris-predictor-default-00001   5h11m

check SERVICE_HOSTNAME, INGRESS_PORT, INGRESS_HOST

SERVICE_HOSTNAME

$ SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n kserve-test -o jsonpath='{.status.url}' | cut -d "/" -f 3)
$ echo $SERVICE_HOSTNAME

sklearn-iris.kserve-test.example.com

INGRESS_PORT

$ INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
$ echo $INGRESS_PORT

INGRESS_HOST
```
$ INGRESS_HOST=192.168.219.100
```
192.168.219.100 : Internal IP of my device

create input

$ cat <<EOF > "./iris-input.json"
{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}
EOF

send request

$ curl -v -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict -d @./iris-input.json

*   Trying 192.168.219.100:31018...
* Connected to 192.168.219.100 (192.168.219.100) port 31018 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kserve-test.example.com
> User-Agent: curl/7.71.1
> Accept: */*
> Content-Length: 76
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY3MTU5MDMxOHxFd3dBRUhZek16WktlRlZJZFc1alowVlROVTA9fFynQ-3082qPF_-qUwnYllySrEQPAKGqpBuF-Pu9gcnx
< date: Wed, 21 Dec 2022 02:38:38 GMT
< x-envoy-upstream-service-time: 6
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host 192.168.219.100 left intact

Got 302 Found.

As far as I know, this result is because I was asked for authentication from Dex.

what i did to solve this

Authentication

I tried to get the authservice_session token by following the method here: kserve:github

From CLI

$ CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')

CLUSTER_IP: 10.103.239.220

$ curl -v http://${CLUSTER_IP}

*   Trying 10.103.239.220:80...
* Connected to 10.103.239.220 (10.103.239.220) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.103.239.220
> User-Agent: curl/7.71.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< content-type: text/html; charset=utf-8
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY3MTU5MzE2MXxFd3dBRUdGM2JUSnlaMkZVYjFWUVNFa3dURGs9fEDuO8ql3cFsetSfKntLvFV0al5tEZJeh23VK-JrJubM
< date: Wed, 21 Dec 2022 03:26:01 GMT
< content-length: 269
< x-envoy-upstream-service-time: 4
< server: istio-envoy
< 
<a href="/dex/auth?client_id=kubeflow-oidc-authservice&amp;redirect_uri=%2Flogin%2Foidc&amp;response_type=code&amp;scope=profile+email+groups+openid&amp;state=MTY3MTU5MzE2MXxFd3dBRUdGM2JUSnlaMkZVYjFWUVNFa3dURGs9fEDuO8ql3cFsetSfKntLvFV0al5tEZJeh23VK-JrJubM">Found</a>.

* Connection #0 to host 10.103.239.220 left intact

I stuck at this stage. I think it's wrong to see 302 Found.

From the browser

Copy the token content from the cookie authservice_session

$ SESSION =MTY3MTUyODQ2M3xOd3dBTkRkRVExbEdVa0kzVFRJMFMwOU9VRE5hV2pSS1VGVkNSRVJVUlRKVlVVOUlTa2hDVWpOU1RUZFRVRkJGVTFGV1N6UktXVkU9fCoQdbMu_diLBJAKLZSmF4qoqQTlINKq7A63hy-QNQcR

$ curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Cookie: authservice_session=${SESSION}"  http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict -d ./iris-input.json

got 404 Not Found

curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Cookie: authservice_session=${SESSION}"  http://${CLUSTER_IP}/v1/models/${MODEL_NAME}:predict -d @./iris-input.json

got 404 Not Found

added Envoy filter for bypass Dex

$ vi envoyfilter.yaml

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: sklearn-iris-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      routeConfiguration:
        vhost:
          name: sklearn-iris.kserve-test.example.com:31018
    patch:
        operation: MERGE
        value:
          per_filter_config:
            envoy.ext_authz:
              disabled: true

$ kubectl apply -f envoyfilter.yaml

Note: spec.configPatches.match.routeConfiguration.vhost.name : sklearn-iris.kserve-test.example.com:31018

It's not working.

Got still 302 Found

External Authorization

reference: https://github.com/kubeflow/kubeflow/issues/4549#issuecomment-932259673

add AuthorizationPolicy

$ vi authorizationpolicy.yaml

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: dex-auth
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: CUSTOM
  provider:
    # The provider name must match the extension provider defined in the mesh config.
    name: dex-auth-provider
  rules:
  # The rules specify when to trigger the external authorizer.
    - to:
      - operation:
          notPaths: ["/v1*"]

$ kubectl apply -f authorizationpolicy.yaml

Note: rules.to.operation: notPaths: ["/v1*"]

and delete envoyfilters named authn-filter that originally existed

$ kubectl delete -n istio-system envoyfilters.networking.istio.io authn-filter

next, restart deployment/istiod

$ kubectl rollout restart deployment/istiod -n istio-system

It's not working.

Got still 302 Found if don't delete envoyfilters named authn-filter that originally existed, or block the connection if I delete authn-filter .

What I need help:

How can I get Dex authorization and make a connected?
Or how can I bypass Dex if I can't get Dex authorization?
Maybe my model serving example is wrong. Thanks for any advice on what's wrong.

env:

ubuntu 20.04

$ kubectl version --client && kubeadm version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:"a43c0904d0de10f92aa3956c74489c45e6453d6e", GitTreeState:"clean", BuildDate:"2022-08-17T18:28:56Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
kubeadm version: &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:

I installed istio and kanative that included in the kubeflow manifest by following: kubeflow/manifasts.

$ kubectl get pod -n istio-system
istio-system                authservice-0                                            1/1     Running
istio-system                cluster-local-gateway-5449f87d9b-bb4vs                   1/1     Running
istio-system                istio-ingressgateway-77b9d69b74-xmv98                    1/1     Running
istio-system                istiod-67fcb675b5-kzfvd                                  1/1     Running

$ kubectl get pod -n knative-eventing  
$ kubectl get pod -n knative-serving
knative-eventing            eventing-controller-8457bd9747-855lc                     1/1     Running
knative-eventing            eventing-webhook-69986cfb5d-hn7tx                        1/1     Running
knative-serving             activator-7c5cd78566-pz6ns                               2/2     Running
knative-serving             autoscaler-98487645d-vh5wk                               2/2     Running
knative-serving             controller-7546f544b7-mng9g                              2/2     Running
knative-serving             domain-mapping-5d56bfc7d-5cb9l                           2/2     Running
knative-serving             domainmapping-webhook-696559d49c-p8rwr                   2/2     Running
knative-serving             net-istio-controller-c4d469c-lt5fl                       2/2     Running 
knative-serving             net-istio-webhook-855bcb6747-wbl4x                       2/2     Running
knative-serving             webhook-59f9fdd446-xsf6n                                 2/2     Running

And installed KServe and KServe Built-in ClusterServingRuntimes by following: kserve installation

$ kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.9.0/kserve.yaml
$ kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.9.0/kserve-runtimes.yaml

$ kubectl get pod -n kserve
kserve-controller-manager-5fc887875d-m7rlp   2/2     Running

check gateway selector

knative-local-gateway in namespace knative-serving

$ kubectl get gateways knative-local-gateway -n knative-serving -o yaml

spec:
  selector:
    app: cluster-local-gateway
    istio: cluster-local-gateway

istio-ingressgateway in namespace istio-system

$ kubectl get gateways istio-ingressgateway -n istio-system -o yaml

spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway

cluster-local-gateway in namespace istio-system

$ kubectl get gateways cluster-local-gateway -n istio-system -o yaml

spec:
  selector:
    app: cluster-local-gateway
    istio: cluster-local-gateway

kubeflow-gateway in namespace kubeflow

$ kubectl get gateways kubeflow-gateway -n kubeflow -o yaml

spec:
  selector:
    istio: ingressgateway

Original Q&A

There are 1 answers

**Neil Wang** · Answer 1 · 2023-03-23 01:47:01

I met the same issue, please try to specify the session by --cookie instead of apply it in the headers.

curl --cookie "authservice_session=${SESSION}" -v -H "Host: ${SERVICE_HOST}" "http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict" -d @./iris-input.json

TechQA.

I run the kserve example for sklearn-iris and got `302 Found`

what i did to solve this

There are 1 answers

Related Questions in KUBEFLOW

Related Questions in KUBEFLOW-KSERVE

Popular Questions

Popular Tags

Trending Questions