I am running HPO jobs in sage maker, and I am thinking of a way to stop my HPO job after one of the child training jobs reaches a specific metrics threshold.
PS: I tried sage maker early stopping, but it only works on the level of epocs within each training job, so it stops training jobs if it noticed that their learning pattern might not give as good metric as the best training jobs found already. But this does not solve my problem which in the level of HPO combinations, so regardless of what happens within the child training jobs, I want to stop the whole HPO job after one of its children reaches my desired metric threshold.