tensorflow model stops early even though val_loss decreases with raytune

16 views Asked by At

Versions:

  • python 3.8
  • tensorflow-cpu 2.12.0
  • ray 2.6.3

I'm training a deep learning model using raytune,

callback_lst = [EarlyStopping(monitor='val_loss', patience=3)]
tuner = tune.Tuner(tune.with_parameters(train_model,..., callback_list=callback_lst))
tuner.fit()


def train_model(config,...,callbak_lst):
    model = instantiate_model()
    history = model.fit(x,validation_data, callbacks=callback_st, epochs=15)

This is one of the log for one of the trained model with sampled hyperprameters

loss     .... val_loss
0.0062         0.0073
0.00498        0.00708
0.0049         0.00729
0.00489        0.007165
0.00488        0.007104

As you can see val_loss don't have 3 consecutive non-descreasing epochs. Yet it stops at 5th epoch. Why is this the case?

0

There are 0 answers