Why does imshow report errors in different locations and VideoWriter.write also report errors

Question

I am using Qt version 6.2.4, and I am making a simple video processing software. I need to take every frame of the video and perform a series of OpenCV operations on that frame. Here is my code to get each frame of the image:

Here is some explanation of my code:

1. processVideoFrame slot function is used to get each frame of the video in response to the signal.
2. I used a std::vector to store every frame of the cv::Mat type, where my QImage was converted to cv::Mat and then push_back into mats
3. In the positionChanged signal, if the time reaches the end of the video, it indicates that the video is finished playing, and then the showMats function is called to show each frame that I have stored before

To make sure it doesn't end immediately, I've added cv::waitKey to pause it. In theory, my showMats function should reproduce the video by going through each Mat, but it's reporting an error at imshow.

class VideoFrameProcessor : public QObject
{
public:
    VideoFrameProcessor(QLabel* label) : videoLabel(label) { 
        mats.clear(); 
        videoWrite.open("video/output.mp4", cv::VideoWriter::fourcc('M', 'J', 'P', 'G'), 25, cv::Size(1566, 1080), true);
    }
    ~VideoFrameProcessor() {
        videoWrite.release();
    }
    void showMats() {
        for (auto& frame : mats) {
            if (frame.empty() || cv::waitKey(10) == 27) {
                break;
            }
            //cv::imshow("1", frame);  //When I use it, I get an error
        }
    }
public slots:
    void processVideoFrame(const QVideoFrame& frame)
    {
        if (frame.isValid())
        {
            QImage image = frame.toImage();
            height = image.height();
            width = image.width();
            cv::Mat mt(image.height(), image.width(), CV_8UC4,image.bits(),image.bytesPerLine());
            mats.push_back(mt);
            cv::imshow("1", mt); // When I use it, it's correct

           //
           //videoWrite.write(mt);
        }
    }
private:
    cv::VideoWriter videoWrite;
    std::vector<cv::Mat> mats; 
    int height, width;
    QLabel* videoLabel;
};

class VideoWidget : public QMainWindow
{
public:
    VideoWidget(QWidget* parent = nullptr) : QMainWindow(parent)
    {
        resize(640, 480);
        QLabel* videoLabel = new QLabel(this);
        videoLabel->show();

        mediaPlayer = new QMediaPlayer(this);
        videoWidget = new QVideoWidget(this);
        mediaPlayer->setVideoOutput(videoWidget);
        videoWidget->show();

        // 设置要播放的视频文件
        mediaPlayer->setSource(QUrl::fromLocalFile("video/cat.mp4"));

        // 播放视频
        mediaPlayer->play();

        // 创建QVideoSink并连接到videoFrameChanged信号
        videoSink = new QVideoSink(this);
        frameProcessor = new VideoFrameProcessor(videoLabel);
        QObject::connect(videoSink, &QVideoSink::videoFrameChanged, frameProcessor, &VideoFrameProcessor::processVideoFrame);
        mediaPlayer->setVideoSink(videoSink);

        connect(mediaPlayer, &QMediaPlayer::positionChanged, this, [=](qint64 val) {
            if (val == mediaPlayer->duration()) {
                //Show a series of Mats
                frameProcessor->showMats();
            }
            });

        this->setCentralWidget(videoWidget);
    }
    
private:
    
    QMediaPlayer* mediaPlayer;
    QVideoWidget* videoWidget;
    QVideoSink* videoSink;
    VideoFrameProcessor* frameProcessor;
};

int main(int argc, char* argv[])
{
    QApplication app(argc, argv);
    VideoWidget w;
    w.show();

    return app.exec();
}

This is an error image after my imshow And output the error code: -1073741819。

As far as I know: std::vector data does not automatically disappear. The data will remain until your program exits or you explicitly release them.

So I also used an imshow in the processVideoFrame function to display the image at any time, and this time it worked. As you can see, this video is of a cat: This is the correct video frame

I want to know why imshow will report errors in different positions?

Answer 1

You get segmentation fault because you use cv::Mat constructor version which only stories a reference to already allocated data i.e. the data created and owned by QImage (we can say that image data are shared between QImage and cv::Mat).

data Pointer to the user data. Matrix constructors that take data and step parameters do not allocate matrix data. Instead, they just initialize the matrix header that points to the specified data, which means that no data is copied. This operation is very efficient and can be used to process external data using OpenCV functions. The external data is not automatically deallocated, so you should take care of it.

Becuase QImage is local, when the code goes out of the scope local QImage is destroyed and cv::Mat holds dangling pointer.

        QImage image = frame.toImage();
        height = image.height();
        width = image.width();
        cv::Mat mt(image.height(), image.width(), CV_8UC4,image.bits(),image.bytesPerLine());
        mats.push_back(mt);

to make the code working, call clone:

mats.push_back(mt.clone());

by clong the deep copy of data shared by mat/qimage is done.