Cityscapes segmentation with pre trained weights

Question

I'm working on images from the Cityscapes dataset that I'm trying to segment with segmentation_models

I have a tensorflow train dataset with around 60 images. Resized to 512 px. I'm trying to compare the performance of my network with and without pre trained weigths and the impact is very limited I'm struggling to understand why.

BACKBONE = 'vgg16'
BATCH_SIZE = 4
CLASSES = CLASSES
LR = 0.0001
EPOCHS = 5
NUM_TRAIN= 60
NUM_VAL = 20
tf.keras.backend.clear_session()
preprocess_input = sm.get_preprocessing(BACKBONE)

n_classes = 1 if len(CLASSES) == 1 else (len(CLASSES))  # case for binary and multiclass segmentation
activation = 'sigmoid' if n_classes == 1 else 'softmax'

model = sm.Unet(BACKBONE, classes=n_classes, activation=activation,encoder_weights='imagenet')
model.summary(show_trainable=True)
optim = keras.optimizers.legacy.Adam(LR)
# train model
mlflow.tensorflow.autolog()
run_description='VGG16 Weights Image Net, 60 train 20 val'
with mlflow.start_run(description=run_description) as run:
    history = model.fit(
        train_dataset_tf, 
        steps_per_epoch=train_dataset_tf.cardinality().numpy(), 
        epochs=EPOCHS, 
        callbacks=callbacks, 
        validation_data=val_dataset_tf, 
        validation_steps=val_dataset_tf.cardinality().numpy()
    )

When I use encoder_weights='imagenet' I got Last Train IOU : 0.15 Last Val IOU : 0.04 Mean Test IOU : 0.15

When I use encoder_weights=None I got Last Train IOU : 0.31 Last Val IOU : 0.09 Mean Test IOU : 0.27

Is there something I am missing ? How can I check that the pre-trained weights are correctly loaded / used ?

Thank you very much in advance !

I tried to freeze the encoder / unfreeze it and the results I get are incoherent :

Detailed résults :

Freeze_encoder = False :

encoder_weights='imagenet' Last Train IOU : 0.26 Last Val IOU : 0.08 Mean Test IOU : 0.15

encoder_weights=None Last Train IOU : 0.31 Last Val IOU : 0.09 Mean Test IOU : 0.27

Encoder_freeze = True,

encoder_weights='imagenet'

Train IOU : 0.11

Val IOU : 0.04

Test : 0.08

encoder_weights='None'

Train : 0.3

Val : 0.05

Test : 0.14