`val_map.txt` not found in ImageNet2012 validation dataset when using the MLPerf Benchmark

167 views Asked by At

I am a complete beginner in running or using ML workloads. I am just following the ReadMe instructions in the corresponding benchmark git pages.

I want to run the MLPerf Inference benchmark suite on resnet50-v1.5 onnx model, using the ImageNet-2012 validation dataset.

enter image description here

I extract the .tar file downloaded from the link above. The .tar file contains no directory structure; it just contains many images. (I thought it be grouped in a specific directory structure to represent the classes).

$ tar -xf  ILSVRC2012_img_val.tar 
$ ls
ILSVRC2012_val_00000001.JPEG  ILSVRC2012_val_00012501.JPEG  ILSVRC2012_val_00025001.JPEG  ILSVRC2012_val_00037501.JPEG
ILSVRC2012_val_00000002.JPEG  ILSVRC2012_val_00012502.JPEG  ILSVRC2012_val_00025002.JPEG  ILSVRC2012_val_00037502.JPEG
ILSVRC2012_val_00000003.JPEG  ILSVRC2012_val_00012503.JPEG  ILSVRC2012_val_00025003.JPEG  ILSVRC2012_val_00037503.JPEG
ILSVRC2012_val_00000004.JPEG  ILSVRC2012_val_00012504.JPEG  ILSVRC2012_val_00025004.JPEG  ILSVRC2012_val_00037504.JPEG
...

I exported the data directory and model directory and then decided to use Docker container for experimentation.

$ ./run_and_time.sh onnxruntime resnet50 gpu

The above command successfully builds the docker image and tries to launch the application inside the container, but the application complains about a missing val_map.txt.

[+] Building 0.9s (16/16) FINISHED                                                                                             
 => [internal] load build definition from Dockerfile.gpu
...
             TRUNCATED DOCKER BUILD OUTPUT
...                                                               
 => => naming to docker.io/library/mlperf-infer-imgclassify-gpu                                                           0.0s 
Clearing caches.                                                                                                               
3
STARTING RUN AT 2023-07-27 07:23:05 PM
INFO:main:Namespace(accuracy=False, audit_conf='audit.config', backend='onnxruntime', cache=0, cache_dir=None, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/home/abhishek/shweta_machine/DGSF_workloads/Image_Classification/ImageNet_2012_validation', debug=False, find_peak_performance=False, inputs=None, max_batchsize=32, max_latency=None, mlperf_conf='./mlperf.conf', model='/home/abhishek/shweta_machine/DGSF_workloads/Image_Classification/resnet50_v1.onnx', model_name='resnet50', output='/output', outputs=['ArgMax:0'], performance_sample_count=None, preprocessed_dir=None, profile='resnet50-onnxruntime', qps=None, samples_per_query=8, scenario='SingleStream', threads=16, time=None, use_preprocessed_dataset=False, user_conf='user.conf')
Traceback (most recent call last):
  File "python/main.py", line 624, in <module>
    main()
  File "python/main.py", line 501, in main
    **kwargs)
  File "/mlperf/python/imagenet.py", line 53, in __init__
    with open(image_list, 'r') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/abhishek/shweta_machine/DGSF_workloads/Image_Classification/ImageNet_2012_validation/val_map.txt'
ENDING RUN AT 2023-07-27 07:23:05 PM
abhishek@abhishek:classification_and_detection$

Looking into the specific section of the codebase I found the following:

        ...
        # input images are in HWC
        self.need_transpose = True if image_format == "NCHW" else False

        self.not_found = 0
        if image_list is None:
            # by default look for val_map.txt
            image_list = os.path.join(data_path, "val_map.txt")  
        with open(image_list, 'r') as fp:                           # <--- line 53
            for count, line in enumerate(fp):
                pass
        count = count + 1
        if not self.count:
            CNT = count
        else:
            CNT = count if count <= self.count else self.count

        os.makedirs(self.cache_dir, exist_ok=True)
        ...

I assume the val_map.txt is required for getting the mapping of class, which is kind of mentioned in here.

But I do not have much knowledge regarding this. Please, can anyone help me?

0

There are 0 answers