Why does Tensorflow 1.11.0 return CUDA_ERROR_NOT_SUPPORTED?

Question

My machine is Ubuntu 18.04.1 LTS, with CUDA has been successfully installed. The output of $nvcc --version is

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

I have two GPUs of Tesla K80, and the command nvidia-smi shows:

output of nvidia-smi

I also tried to test with ./deviceQuery from NVIDIA_CUDA-9.0_Samples and its output is as the followings:

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 2 CUDA Capable device(s)`

...

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.0, CUDA Runtime Version = 9.0, NumDevs = 2
Result = PASS

However, once I install Tensorflow GPU version 1.11.0 from pip, I couldn't open a Tensorflow session.

>>> import tensorflow as tf
>>> sess = tf.Session()

and it outputs:

2018-11-15 00:13:46.593039: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/quoctin.phan/tools/anaconda/envs/tensorflow_1.11/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1511, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/home/quoctin.phan/tools/anaconda/envs/tensorflow_1.11/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 634, in __init__
    self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_NOT_SUPPORTED: operation not supported

I have tried to reinstall Tensorflow 1.12.0, but nothing changes. Your help is appreciated.

Answer 1

Do you think your problem might be somehow connected to Compute Capability? The problem is described in here.

You can check them when you run deviceQuery.exe. Here is a thread about where to find it on windows distribution of CUDA package.