nvprof Warning: The path to CUPTI and CUDA Injection libraries might not be set in LD_LIBRARY_PATH

1.2k views Asked by At

I get the message in the subject when I try to run a program I developed with OpenACC through Nvidia's nvprof profiler like this:

nvprof ./SFS 4

If I run nvprof with -o [output_file] the warning message doesn't appear, but the output file is not created. What could be wrong here?

The LD_LIBRARY_PATH is set in my .bashrc to: /opt/nvidia/hpc_sdk/Linux_x86_64/20.7/cuda/11.0/lib64/ because there I have found these files there (they have "cupti" and "inj" in their names and I thought they are the needed ones):

lrwxrwxrwx 1 root root      19 Aug  4 05:27 libaccinj64.so -> libaccinj64.so.11.0
lrwxrwxrwx 1 root root      23 Aug  4 05:27 libaccinj64.so.11.0 -> libaccinj64.so.11.0.194
...
lrwxrwxrwx 1 root root      16 Aug  4 05:27 libcupti.so -> libcupti.so.11.0
lrwxrwxrwx 1 root root      20 Aug  4 05:27 libcupti.so.11.0 -> libcupti.so.2020.1.0
...

I am on Ubuntu 18.04. workstation with Nvidia GeForce RTX 2070, and have CUDA version 11 installed.

nvidia-smi command gives me this:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66       Driver Version: 450.66       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:02:00.0  On |                  N/A |
| 30%   40C    P2    58W / 185W |    693MiB /  7981MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

The compilers I have (nvidia and portland) are from the latest Nvidia HPC-SDK, version 20.7-0

I compile my programs with -acc -Minfo=accel options, not sure how could I set -ta= and if it is needed at all?

P.S. I am also not sure if running my code, with or without nvprof uses GPUs at all, although I did set ACC_DEVICE_TYPE to nvidia.

Any advice would be very welcome.

Cheers

1

There are 1 answers

5
Mat Colgrove On

Which nvprof are you using? The one that ships with NV HPC 20.7 or your own install?

This looks very similar to an issue reported yesterday on the NVIDIA DevTalk user forums:

https://forums.developer.nvidia.com/t/new-20-7-version-where-is-the-detail-release-bugfix/146168/4

Granted this was for Nsight-systems, but it may be the same issue. It appears to be a problem with the 2020.3 version of the profilers which is the version we ship with the NV HPC 20.7 SDK. As I note, the Nsight-Systems 2020.4 release should have this fixed, so the work around would be download and install 2020.4 or use a prior release.

https://developer.nvidia.com/nsight-systems

There does seem to be a temporary issue with the Nsight-systems download that hopefully be corrected before you see this note.

Also, nvprof is in the process of being deprecated so you should consider moving to use Nsight-systems and Nsight-compute.

https://developer.nvidia.com/blog/migrating-nvidia-nsight-tools-nvvp-nvprof/