CUDNN_STATUS_EXECUTION_FAILED error in tensorflow model on GPU

Question

I am trying to compile a tensorflow model using UNET architecture (OS->Rocky Linux 8.6, GPU->Quadro P620, Tensoflow-> 2.11.0, CUDA->11.6). The model works fine on CPU and google colab. But when i try to run it on GPU then the following problem comes during model.fit.

CUDNN_STATUS_EXECUTION_FAILED in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5750): 'cudnnBatchNormalizationForwardTrainingEx( cudnn.handle(), mode, bn_ops, &one, &zero, x_descriptor.handle(), x.opaque(), x_descriptor.handle(), side_input.opaque(), x_descriptor.handle(), y->opaque(), scale_offset_descriptor.handle(), scale.opaque(), offset.opaque(), exponential_average_factor, batch_mean_opaque, batch_var_opaque, epsilon, saved_mean->opaque(), saved_inv_var->opaque(), activation_desc.handle(), workspace.opaque(), workspace.size(), reserve_space.opaque(), reserve_space.size())'

Answer 1

This is because you have installed CUDA 11.6 for Tensorflow 2.11. You need to install CUDA 11.2 and cuDNN 8.1 as mentioned in the below tested build configurations for linux:

Please follow all the step by step instructions mentioned in this link along with the specified Hardware and Software requirements to enable tensorflow with GPU Support in your system.