os.listdir() in Python when listing files from network drive uses too much memory/takes long time

Question

I am trying to use os.listdir('somefilepath') to list files in a network drive. Despite the fact the number of files is not too large (around 3000 small files in the directory), the task just takes forever/does not finish. When I look at task manager, python takes ever increasing amount of memory to complete the task.

I have tried using os.walk instead, an glob.glob/glob.iglob functions but it does not help. Is there an issue with the network latency? What is the best way to check?

Answer 1

Try using scandir which is available in Python 3. It should perform better.

code might look something like this:

import os

file_list = os.scandir('your_dir_path')

see scandir 1.3 and PEP 471