Slack webhook and model training tasks

Question

This should be a fairly simple question but I haven't be able to find anything in the documentation: is there a way to send a Slack notification when a training task (in GCP ai platform) gets completed? Basically I am looking for the AI platform eqiivalent of this.

Answer 1

For sending a Slack notification when a training task gets completed, a possible solution might be setting up a Pub/Sub sink for matching logs from Cloud Logging to be sent to. Details here in Exporting logs with the Google Cloud Console. If a log matches the sink’s query (look for all training status updates from AI Platform), then Cloud Logging will send the log directly to the indicated Pub/Sub topic and extra VM is not needed for this.

After sending the messages to a Pub/Sub topic, you can set up a Cloud Function to post messages to Slack (and/or to email you) when the logs indicate that the AI Platform training job is done. Please note that the Cloud Function will only run when a relevant log is posted to Pub/Sub.Details on Google Cloud Pub/Sub Triggers and Configuring Slack notifications.

In order to interpret what the log received from Cloud Logging means (your training has completed), you can read the text associated with each log in the JSON payload message. Cloud Functions would not need to make API calls to ml.googleapis.com in this situation.Here I am attaching Cloud Function Pricing for reference only.

Answer 2

Today, the only way is to perform a filter in the logs and to sink them into PubSub. Then, your pubsub message can trigger the Cloud Run container deployed with the Slack Notifier, as described in the tutorial.

Soon, it will be possible to simply define the event and to call directly the CLoud Run service without performing a sink into PubSub, but it's only in Alpha for now.