When Using YOLOv8 Detector with BoT-SORT I see warning related to optical flow

Question

Warning with gmc.py, line 273, not enough matching point. I'm using the BoT-SORT tracking model using the YOLOv8 detection model. For some videos, I see this warning: "WARNING: not enough matching points" and after that, I see this:

Traceback (most recent call last):
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/yolov8_track_third_cell.py", line 44, in <module>
    results = model.track(frame, persist=True, show=False, tracker="botsort.yaml", conf= 0, iou = 0.5)  # predict on an image
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/ultralytics/ultralytics/engine/model.py", line 262, in track
    return self.predict(source=source, stream=stream, **kwargs)
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/ultralytics/ultralytics/engine/model.py", line 242, in predict
    return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/ultralytics/ultralytics/engine/predictor.py", line 196, in __call__
    return list(self.stream_inference(source, model, *args, **kwargs))  # merge list of Result into one
  File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
    response = gen.send(None)
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/ultralytics/ultralytics/engine/predictor.py", line 264, in stream_inference
    self.run_callbacks('on_predict_postprocess_end')
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/ultralytics/ultralytics/engine/predictor.py", line 358, in run_callbacks
    callback(self)
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/ultralytics/ultralytics/trackers/track.py", line 48, in on_predict_postprocess_end
    tracks = predictor.trackers[i].update(det, im0s[i])
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/ultralytics/ultralytics/trackers/byte_tracker.py", line 278, in update
    warp = self.gmc.apply(img, dets)
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/ultralytics/ultralytics/trackers/utils/gmc.py", line 85, in apply
    return self.applySparseOptFlow(raw_frame, detections)
  File "/media/strh/MyDrive/Track/vision-YOLOv8_Evolve/vision/ultralytics/ultralytics/trackers/utils/gmc.py", line 273, in applySparseOptFlow
    matchedKeypoints, status, err = cv2.calcOpticalFlowPyrLK(self.prevFrame, frame, self.prevKeyPoints, None)
cv2.error: OpenCV(4.6.0) /croot/opencv-suite_1676452025216/work/modules/video/src/lkpyramid.cpp:1260: error: (-215:Assertion failed) (npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0 in function 'calc'

Please help me to solve the problem.

Answer 1

Please add code snippets and more context to your question for viewers

This issue in my case was caused by low confidence detections, with a confidence score lower than that of the track_high_thresh, being passed to the tracker.

See this pull request for more details.

I found two potential fixes:

1. Set the confidence score of your predictions higher (model.track(source="my-source", tracker='botsort.yaml', show=True, conf=0.4)

2. Add the following at line 327 of ultralytics/trackers/utils/gmc.py. (This assumes you are up for editing the ultralytics package)

        if self.prevKeyPoints is None:
            self.prevFrame = frame.copy()
            self.prevKeyPoints = copy.copy(keypoints)
            return H