Ref - https://kserve.github.io/website/master/admin/kubernetes_deployment/
kind - kubernetes v1.25 Istio - v1.16.0 Cert Manager - v1.14.2 kserve v0.11.0 (deployed kserve without knative)
on my kind k8s cluster.
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-cainjector-76bbdd77f7-w4rpk 1/1 Running 4 (106m ago) 22h
cert-manager cert-manager-cdbc489b6-mnq7d 1/1 Running 5 (107m ago) 22h
cert-manager cert-manager-webhook-7ffbff4575-5tlrb 1/1 Running 4 (106m ago) 22h
istio-system istio-ingressgateway-748fb66b49-2w2wf 1/1 Running 2 (107m ago) 22h
istio-system istiod-5d74c58fdd-x4br5 1/1 Running 2 (107m ago) 22h
kserve kserve-controller-manager-55d7c5685f-7s75b 2/2 Running 6 (107m ago) 22h
kserve sklearn-iris-example-predictor-776df85f86-n6xvd 1/1 Running 118 (6m5s ago) 21h
kube-system coredns-565d847f94-7f7nn 1/1 Running 2 (107m ago) 22h
kube-system coredns-565d847f94-vqvvk 1/1 Running 2 (107m ago) 22h
kube-system etcd-ethan-control-plane 1/1 Running 2 (107m ago) 22h
kube-system kindnet-jj7v7 1/1 Running 3 (107m ago) 22h
kube-system kube-apiserver-ethan-control-plane 1/1 Running 2 (107m ago) 22h
kube-system kube-controller-manager-ethan-control-plane 1/1 Running 2 (107m ago) 22h
kube-system kube-proxy-vzbkb 1/1 Running 2 (107m ago) 22h
kube-system kube-scheduler-ethan-control-plane 1/1 Running 2 (107m ago) 22h
local-path-storage local-path-provisioner-684f458cdd-8d4qf 1/1 Running 4 (106m ago) 22h
kubectl -n kserve logs -f sklearn-iris-example-predictor-776df85f86-n6xvd
INFO:root:Copying contents of /mnt/models to local
ERROR:root:fail to locate model file for model sklearn-iris-example under dir /mnt/models,trying loading from model repository.
Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/sklearnserver/sklearnserver/__main__.py", line 42, in <module>
kserve.ModelServer(registered_models=SKLearnModelRepository(args.model_dir)).start(
File "/sklearnserver/sklearnserver/sklearn_model_repository.py", line 24, in __init__
self.load_models()
File "/kserve/kserve/model_repository.py", line 37, in load_models
for name in os.listdir(self.models_dir):
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/models'
Similar issue ref - https://github.com/kserve/kserve/issues/3082
Anyone face a similar issue and have a solution or workaround?
Thank you