How can I calculate Intersection Over Union in Detectron2?

Question

I am using Detectron2 for object detection. I have registered pascalvoc dataset and trained a model for detection. How can I calculate Mean IOU of my test dataset ? I know that detection2 has a predefined function for calculating IOU i.e. detectron2.structures.pairwise_iou

I have the ground truth bounding boxes for test images in a csv file. The csv file contains (filename,width,height,class,xmin,ymin,xmax,ymax). How can I parse both the bounding boxes in IOU function and display it in google colab.

This is my code where I am generating a prediction bounding box

from detectron2.utils.visualizer import ColorMode
import random

dataset_dicts = DatasetCatalog.get('/content/test')
for d in random.sample(dataset_dicts, 5):    
    im = cv2.imread(d["file_name"])
    outputs = predictor(im)
    v = Visualizer(im[:, :, ::-1], metadata=microcontroller_metadata, scale=0.8)
    v = v.draw_instance_predictions(outputs["instances"].to("cpu"))
    plt.figure(figsize = (14, 10))
    plt.imshow(cv2.cvtColor(v.get_image()[:, :, ::-1], cv2.COLOR_BGR2RGB))
    plt.show()

Answer 1

You need to do what Alexander said. So let's say that for a given image you have the ground truth boxes. These you can be represented as an N,4 numpy array. Let's call it bboxes_gt. and let's say your prediction gives you M bounding boxes.

Now you can convert bboxes_gt into a Box oobject and then use structures.pairwise_iou() to compute all IOUs in a pariwise fashion. This will give you an N,M matrix with all these IOUs. It would look something like this:

bboxes_gt = structures.Boxes(torch.Tensor(bboxes_gt))
bboxes_pred = outputs["instances"].pred_boxes
IOUs = structures.pairwise_iou(bboxes_gt, bboxes_pred)

Answer 2

You have access to the bounding boxes and the classes via

outputs["instances"].pred_boxes.tensor.cpu().numpy()
outputs["instances"].pred_classes.cpu().numpy()

Now what you have to do:

1. Do your inferencing on the picture
2. Load the ground truth for the picture from your csv file
3. Compare the classes and bounding boxes of your inference with your ground trouth (= IoU)