I use theano for some deep learning experiments. I have killed a 3 weeks running process by ctrl+c, to start a new process.
As I see, although I have killed the process, the gpu memory is not released. According to nvidia-smi, the memory is free, except 23MB small usage. I use Tesla k40.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.57 Driver Version: 367.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K40m Off | 0000:85:00.0 Off | 0 |
| N/A 24C P8 21W / 235W | 23MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2873 G /usr/lib/xorg/Xorg 23MiB |
+-----------------------------------------------------------------------------+
But in reality, when I try to run even very small datasets, I get memory error. If it would be only 23 MB usage, it shouldn't be a problem at all.
I don't have sudo privileges on the machine I am using. How can I fix this problem?