User Managed Notebook Auto Shut Down

Question

I have been researching to find a suitable and scalable functionality accross multiple project to auto-shutdown user managed notebook in Vertex AI to reduce the idle costs. I saw that "official" solution for this feature is only available for managed notebooks(https://cloud.google.com/vertex-ai/docs/workbench/user-managed/shut-down). In past some concerns has been raised for the similar issue (https://issuetracker.google.com/issues/237305892) which was marked as won't fixed.

Does anyone have any recommendation to establish a auto-shutdown policy for vertexAI user managed notebooks, any suggestions or workarounds are highly appreciated.

I have seen there are some functionalities available to use python/shell script for brute forcing to shutdown VM but as I said my concern of using reliable solutions at production level

Answer 1

The official answer is to use Workbench Instances. Workbench instances include idle-shutdown functionality. Workbench instances are now available in Public Preview.

In the past we had an script that used CPU utilization, but is not recommended anymore as now Idle Shutdown logic is tied to Notebook execution activity.

Answer 2

There is now a clickops button in the user managed notebook gui, under the notebook's "Software and Security" tab, "Enable Idle Shutdown".

From our investigation, all this does is add a key value pair in the notebook metadata, "idle-timeout-seconds" (with a value in seconds for the shutdown to occur).

Modifying the metadata via terraform or cli commands can set this value, and doing it this way activates the GUI button with the value provided, and idle shutdown functions correctly.

This isn't really any official documentation on this though, so theoretically things may change at some stage.