I am trying to use Google Vertex AI Prediction model. I try to predict a data point using the below code.
from google.cloud import aiplatform
project_id = "project_id"
location = "us-central1"
endpoint_id = "399999999999999999"
ai_ep_client_options = {"api_endpoint": "us-central1-aiplatform.googleapis.com"}
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "service-key.json"
instance_list = [{"feature-1":"apple", "feature-2": "banana"}]
aiplatform_client = aiplatform.gapic.PredictionServiceClient(client_options=ai_ep_client_options)
prediction_endpoint_client = aiplatform_client.endpoint_path(project=project_id, location=location, endpoint=endpoint_id)
response = aiplatform_client.predict(endpoint=prediction_endpoint_client, instances=instance_list)
I am getting the below error:
_InactiveRpcError Traceback (most recent call last) File ~\AppData\Local\anaconda3\Lib\site-packages\google\api_core\grpc_helpers.py:50, in wrap_unary_errors..error_remapped_callable(*args, **kwargs) 49 try: ---> 50 return callable(*args, **kwargs) 51 except grpc.RpcError as exc:
File ~\AppData\Local\anaconda3\Lib\site-packages\grpc_channel.py:1161, in _UnaryUnaryMultiCallable.call(self, request, timeout, metadata, credentials, wait_for_ready, compression) 1155 ( 1156 state, 1157 call, 1158 ) = self._blocking( 1159 request, timeout, metadata, credentials, wait_for_ready, compression 1160 ) -> 1161 return _end_unary_response_blocking(state, call, False, None)
File ~\AppData\Local\anaconda3\Lib\site-packages\grpc_channel.py:1004, in _end_unary_response_blocking(state, call, with_call, deadline) 1003 else: -> 1004 raise _InactiveRpcError(state)
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:142.250.31.95:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:xxx.xxx.xxx.x:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED {created_time:"2023-11-03T11:03:27.2979758+00:00", grpc_status:14}"
The above exception was the direct cause of the following exception:
ServiceUnavailable Traceback (most recent call last) Cell In[18], line 16 13 aiplatform_client = aiplatform.gapic.PredictionServiceClient(client_options=ai_ep_client_options) 14 prediction_endpoint_client = aiplatform_client.endpoint_path(project=project_id, location=location, endpoint=endpoint_id) ---> 16 response = aiplatform_client.predict(endpoint=prediction_endpoint_client, instances=instance_list) 17 response
File ~\AppData\Local\anaconda3\Lib\site-packages\google\cloud\aiplatform_v1\services\prediction_service\client.py:602, in PredictionServiceClient.predict(self, request, endpoint, instances, parameters, retry, timeout, metadata) 597 metadata = tuple(metadata) + ( 598 gapic_v1.routing_header.to_grpc_metadata((("endpoint", request.endpoint),)), 599 ) 601 # Send the request. --> 602 response = rpc( 603 request, 604 retry=retry, 605 timeout=timeout, 606 metadata=metadata, 607 ) 609 # Done; return the response. 610 return response
File ~\AppData\Local\anaconda3\Lib\site-packages\google\api_core\gapic_v1\method.py:154, in _GapicCallable.call(self, timeout, retry, *args, **kwargs) 151 metadata.extend(self._metadata) 152 kwargs["metadata"] = metadata --> 154 return wrapped_func(*args, **kwargs)
File ~\AppData\Local\anaconda3\Lib\site-packages\google\api_core\grpc_helpers.py:52, in wrap_unary_errors..error_remapped_callable(*args, **kwargs) 50 return callable(*args, **kwargs) 51 except grpc.RpcError as exc: ---> 52 raise exceptions.from_grpc_error(exc) from exc
ServiceUnavailable: 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:xxx.xx.xx.xx:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
I am running this on "Windos" OS.
aiplatform.__version__
returns 1.28.1.
I am running this code on my corporate machine. This code used to run successfully few days ago. But there are some security updates done on the systems and since then this error started coming up. I only a ML/Python expert, and i dont know the basics of Networking, security certs, CA files, .pem files etc..,
In my research, I have seen some solutions to similar problems, but they were mostly on Linux/Unix systems. Suggest some solutions on Windows.
I am not sure if this is a Networking/Security related Issue or a GCP Python Vertex AI Issue. Kindly excuse if the tags are ir-relevant.
First thing that comes to mind is Firewall. Since you mentioned that there was a security update prior the issue, it might have reconfigured your network firewall blocking the connection to your Vertex AI service.
Troubleshoot your network connection to ensure that its not intercepting or modifying your SSL handshake, at the same time, check you machine's certificate and make sure it is still valid and has not expired.
Refer to this document to use Private Service Connect to access Vertex AI online predictions from on-premises.
You can also reach out to Vertex AI support for detailed assistance.