saturated contrast and low brightness in tensorboard when training TF2 object_detection API

Question

I'm trying to fine-tune an efficient det model. Here is a recap of what I've done:

• download coco dataset 2014
• convert to tfrecord with a script from tensorflow
• download efficientDet D0 from official model zoo
• edit pipeline.config (batch_size: 1, sync_replicas: false, replicas_to_aggregate: 1, fine_tune_checkpoint_type: "detection", use_bfloat16: false) and adjust the paths.
• clone github.com/tensorflow/models.git, docker-compose run object_detection.
• inside the container:

python models/research/object_detection/model_main_tf2.py \
    --pipeline_config_path=efficientdet_d0_coco17_tpu-32/pipeline.config \
    --model_dir=foo/model/ \
    --alsologtostderr

My problem is that as seen in tensorboard (ie after data preprocessing), contrast is maxed out (or sometimes not maxed, but still way too high), and brightness is often too low:

I checked the content of the tfrecords with https://github.com/sulc/tfrecord-viewer, the colors are fine. I tried on another machine with a different nvidia GPU model, same problem.

Any idea where the problem could come from? Thanks!

Answer 1

this seems to be a visualization issue, and not a training issue. It can be solved by changing the normalization from (-1,1) to (0,1).

follow these changes in the code:

https://github.com/tensorflow/models/pull/9019