On my machine Ubuntu 20.04.4 LTS, the command "nvidia-smi" suddenly does not work, it keeps showing static historical information.
Some programs using GPU have stopped, but when running "nvidia-smi", we can still see the GPU usage. However, when we found the program through PID, it did have been killed.
I run a new program using GPU. But when I ran the command "nvidia-smi", no GPU usage of the new program was shown.
Following is the picture when I run "nvidia-smi" for the first time nvidia-smi result 1
Then the second time nvidia-smi result 2
They are the same, and nvidia-smi does not update any new information.
When I run "watch -n 1 nvidia-smi" to dynamically see the GPU usage, it remains static and displays the wrong GPU usage.
Further, when I ran the command "nvidia-smi -h", help information was not shown, It kept showing the static historical information shown in the above figures.