Uneven Multiple GPUs usage using Tensorflow

32 views Asked by At

I am training a model using tensorflow, somehow only the first two GPU were used heavily, occupying 10000 MiB and 9290 MiB, respectively.

I have eight GPUs, the six other GPUs were used but only 300 MB each. Does anyone know why the GPU usage is not evenly distributed across all GPUs?

I use the following code:


strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = create_unet_model()
    model.compile(...)
    model.fit(...)

I looked it up, but no one could suggest any possible reason.

0

There are 0 answers