TechQA.

I run the kserve example for sklearn-iris and got `302 Found`

829 views Asked by At

serving exam model

  1. create namespace

    $ kubectl create namespace kserve-test

  2. create InferenceService

    $ kubectl apply -n kserve-test -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
EOF

    check

    $ kubectl get inferenceservices sklearn-iris -n kserve-test

    NAME           URL                                           READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION                    AGE
sklearn-iris   http://sklearn-iris.kserve-test.example.com   True           100                              sklearn-iris-predictor-default-00001   5h11m

  3. check SERVICE_HOSTNAME, INGRESS_PORT, INGRESS_HOST

    • SERVICE_HOSTNAME

      $ SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n kserve-test -o jsonpath='{.status.url}' | cut -d "/" -f 3)
$ echo $SERVICE_HOSTNAME

      sklearn-iris.kserve-test.example.com

    • INGRESS_PORT

      $ INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
$ echo $INGRESS_PORT

      31018

    • INGRESS_HOST

      $ INGRESS_HOST=192.168.219.100

      192.168.219.100 : Internal IP of my device

  4. create input

    $ cat <<EOF > "./iris-input.json"
{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}
EOF

  5. send request

    $ curl -v -H "Host: ${SERVICE_HOSTNAME}" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict -d @./iris-input.json

    *   Trying 192.168.219.100:31018...
* Connected to 192.168.219.100 (192.168.219.100) port 31018 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kserve-test.example.com
> User-Agent: curl/7.71.1
> Accept: */*
> Content-Length: 76
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY3MTU5MDMxOHxFd3dBRUhZek16WktlRlZJZFc1alowVlROVTA9fFynQ-3082qPF_-qUwnYllySrEQPAKGqpBuF-Pu9gcnx
< date: Wed, 21 Dec 2022 02:38:38 GMT
< x-envoy-upstream-service-time: 6
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host 192.168.219.100 left intact

    Got 302 Found.

    As far as I know, this result is because I was asked for authentication from Dex.

what i did to solve this

  1. Authentication

    I tried to get the authservice_session token by following the method here: kserve:github

    1. From CLI

      $ CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')

      CLUSTER_IP: 10.103.239.220

      $ curl -v http://${CLUSTER_IP}

      *   Trying 10.103.239.220:80...
* Connected to 10.103.239.220 (10.103.239.220) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.103.239.220
> User-Agent: curl/7.71.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< content-type: text/html; charset=utf-8
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY3MTU5MzE2MXxFd3dBRUdGM2JUSnlaMkZVYjFWUVNFa3dURGs9fEDuO8ql3cFsetSfKntLvFV0al5tEZJeh23VK-JrJubM
< date: Wed, 21 Dec 2022 03:26:01 GMT
< content-length: 269
< x-envoy-upstream-service-time: 4
< server: istio-envoy
< 
<a href="/dex/auth?client_id=kubeflow-oidc-authservice&amp;redirect_uri=%2Flogin%2Foidc&amp;response_type=code&amp;scope=profile+email+groups+openid&amp;state=MTY3MTU5MzE2MXxFd3dBRUdGM2JUSnlaMkZVYjFWUVNFa3dURGs9fEDuO8ql3cFsetSfKntLvFV0al5tEZJeh23VK-JrJubM">Found</a>.

* Connection #0 to host 10.103.239.220 left intact

      I stuck at this stage. I think it's wrong to see 302 Found.

    2. From the browser

      Copy the token content from the cookie authservice_session

      $ SESSION =MTY3MTUyODQ2M3xOd3dBTkRkRVExbEdVa0kzVFRJMFMwOU9VRE5hV2pSS1VGVkNSRVJVUlRKVlVVOUlTa2hDVWpOU1RUZFRVRkJGVTFGV1N6UktXVkU9fCoQdbMu_diLBJAKLZSmF4qoqQTlINKq7A63hy-QNQcR

      $ curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Cookie: authservice_session=${SESSION}"  http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict -d ./iris-input.json

      got 404 Not Found

      curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Cookie: authservice_session=${SESSION}"  http://${CLUSTER_IP}/v1/models/${MODEL_NAME}:predict -d @./iris-input.json

      got 404 Not Found

  2. added Envoy filter for bypass Dex

    $ vi envoyfilter.yaml

    apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: sklearn-iris-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      routeConfiguration:
        vhost:
          name: sklearn-iris.kserve-test.example.com:31018
    patch:
        operation: MERGE
        value:
          per_filter_config:
            envoy.ext_authz:
              disabled: true

    $ kubectl apply -f envoyfilter.yaml

    Note: spec.configPatches.match.routeConfiguration.vhost.name : sklearn-iris.kserve-test.example.com:31018

    It's not working.

    Got still 302 Found

  3. External Authorization

    reference: https://github.com/kubeflow/kubeflow/issues/4549#issuecomment-932259673

    add AuthorizationPolicy

    $ vi authorizationpolicy.yaml

    apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: dex-auth
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: CUSTOM
  provider:
    # The provider name must match the extension provider defined in the mesh config.
    name: dex-auth-provider
  rules:
  # The rules specify when to trigger the external authorizer.
    - to:
      - operation:
          notPaths: ["/v1*"]

    $ kubectl apply -f authorizationpolicy.yaml

    Note: rules.to.operation: notPaths: ["/v1*"]

    and delete envoyfilters named authn-filter that originally existed

    $ kubectl delete -n istio-system envoyfilters.networking.istio.io authn-filter

    next, restart deployment/istiod

    $ kubectl rollout restart deployment/istiod -n istio-system

    It's not working.

    Got still 302 Found if don't delete envoyfilters named authn-filter that originally existed, or block the connection if I delete authn-filter .

What I need help:

  • How can I get Dex authorization and make a connected?
  • Or how can I bypass Dex if I can't get Dex authorization?
  • Maybe my model serving example is wrong. Thanks for any advice on what's wrong.

env:

  • ubuntu 20.04

  • $ kubectl version --client && kubeadm version

    Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:"a43c0904d0de10f92aa3956c74489c45e6453d6e", GitTreeState:"clean", BuildDate:"2022-08-17T18:28:56Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
kubeadm version: &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:

  • I installed istio and kanative that included in the kubeflow manifest by following: kubeflow/manifasts.

    $ kubectl get pod -n istio-system
istio-system                authservice-0                                            1/1     Running
istio-system                cluster-local-gateway-5449f87d9b-bb4vs                   1/1     Running
istio-system                istio-ingressgateway-77b9d69b74-xmv98                    1/1     Running
istio-system                istiod-67fcb675b5-kzfvd                                  1/1     Running

    $ kubectl get pod -n knative-eventing  
$ kubectl get pod -n knative-serving
knative-eventing            eventing-controller-8457bd9747-855lc                     1/1     Running
knative-eventing            eventing-webhook-69986cfb5d-hn7tx                        1/1     Running
knative-serving             activator-7c5cd78566-pz6ns                               2/2     Running
knative-serving             autoscaler-98487645d-vh5wk                               2/2     Running
knative-serving             controller-7546f544b7-mng9g                              2/2     Running
knative-serving             domain-mapping-5d56bfc7d-5cb9l                           2/2     Running
knative-serving             domainmapping-webhook-696559d49c-p8rwr                   2/2     Running
knative-serving             net-istio-controller-c4d469c-lt5fl                       2/2     Running 
knative-serving             net-istio-webhook-855bcb6747-wbl4x                       2/2     Running
knative-serving             webhook-59f9fdd446-xsf6n                                 2/2     Running

  • And installed KServe and KServe Built-in ClusterServingRuntimes by following: kserve installation

    $ kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.9.0/kserve.yaml
$ kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.9.0/kserve-runtimes.yaml

    $ kubectl get pod -n kserve
kserve-controller-manager-5fc887875d-m7rlp   2/2     Running

  • check gateway selector

    • knative-local-gateway in namespace knative-serving

      $ kubectl get gateways knative-local-gateway -n knative-serving -o yaml

      spec:
  selector:
    app: cluster-local-gateway
    istio: cluster-local-gateway

    • istio-ingressgateway in namespace istio-system

      $ kubectl get gateways istio-ingressgateway -n istio-system -o yaml

      spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway

    • cluster-local-gateway in namespace istio-system

      $ kubectl get gateways cluster-local-gateway -n istio-system -o yaml

      spec:
  selector:
    app: cluster-local-gateway
    istio: cluster-local-gateway

    • kubeflow-gateway in namespace kubeflow

      $ kubectl get gateways kubeflow-gateway -n kubeflow -o yaml

      spec:
  selector:
    istio: ingressgateway
kubeflow kubeflow-kserve
Original Q&A
1

There are 1 answers

0
Neil Wang On

I met the same issue, please try to specify the session by --cookie instead of apply it in the headers.

curl --cookie "authservice_session=${SESSION}" -v -H "Host: ${SERVICE_HOST}" "http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict" -d @./iris-input.json

Related Questions in KUBEFLOW

Related Questions in KUBEFLOW-KSERVE

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Trending Questions