I have a Python script that uses EasyOCR and Transformers to parse images. After running the script multiple times, the initialization process becomes slow and the script takes longer to start. The only way to restore optimal performance is to reboot the server or reset the Nvidia module using the commands "rmmod nvidia" and "modprobe nvidia".
I suspect that the slow initialization is related to the fragmentation of GPU memory, but I'm not sure how to address this issue. I have tried manually releasing memory using the "cuda.empty_cache()" method, but this does not seem to have any effect. Killing processes visible via "nvidia-smi" and retrying also doesn't help.
The slow initialization is observed exactly after n runs , n = Total Gpu memory/memoryUsedBySingleRun
Has anyone experienced a similar issue with their Python scripts and know how to resolve it? Are there any best practices for managing GPU memory fragmentation that I should be aware of? Any guidance or advice would be greatly appreciated.
Hardware: GRID RTX8000P-24Q Drivers: 470.129.06 CUDA: 11.4 OS: Ubuntu 22.04
Thank you in advance.