A bit of context:
I am writing a media player where I have to draw a big point cloud and some more geometric shapes on any given video. The complexity is also that we don't have a defined format of video and it can come in any format. (fps, resolution)
A limitation is that we need to use QT specific components and any external library would be under discussion if there is no option in QT.
I did implemented a QAbstractVideoSurface
to read frames and then edit them with dynamic data. I have two specific issues with the later mentioned code:
- The point where image is loaded into
QPainter
, it becomes very slow depending upon the resolution of the image. At low resolution the performance is just as good as real-time but at high resolution its really bad.
First of all, I know this is a known issue and widely discussed but none of the answers I found are suitable for me as all of them end up in some case specific solution.
- Even in the cases where i get video drawn at almost real-time, the rectangles I am drawing are missing edges frequently when displayed as video. A pure dump of images show every pixel properly plotted.
I must accept that I really did not knew anything about media processing until I had this task and since everything is to happening on the fly, I didn't had time to read much of text. However, if you point me to the correct document, I would go through it from begging to end. :)
Now below is my code for present method of QAbstractVideoSurface
implementation:
bool VideoFrameGrabber::present(const QVideoFrame &frame)
{
if (frame.isValid())
{
QVideoFrame cloneFrame(frame);
cloneFrame.map(QAbstractVideoBuffer::ReadOnly);
qDebug()<<"1 "<<QTime::currentTime().msecsSinceStartOfDay();
const QImage image(cloneFrame.bits(),
cloneFrame.width(),
cloneFrame.height(),
QVideoFrame::imageFormatFromPixelFormat(cloneFrame.pixelFormat()));
qDebug()<<"2 "<<QTime::currentTime().msecsSinceStartOfDay();
imageEdited = image;
qDebug()<<"3 "<<QTime::currentTime().msecsSinceStartOfDay();
QPainter p(&imageEdited);
qDebug()<<"4 "<<QTime::currentTime().msecsSinceStartOfDay();
if(rectMetaData.count() > 0){
while(rectMetaData.at(rectSequence).frameNumber == frameSequence && rectSequence < rectCount-1)
{
if(rectMetaData.at(rectSequence).fillStatus){ p.setBrush(QBrush(QColor(rectMetaData.at(rectSequence).color)));
} else {
p.setPen(rectMetaData.at(rectSequence).color);
}
p.drawRect(rectMetaData.at(rectSequence).x1,
rectMetaData.at(rectSequence).y1,
rectMetaData.at(rectSequence).x2,
rectMetaData.at(rectSequence).y2);
rectSequence++;
}
}
qDebug()<<"5 "<<QTime::currentTime().msecsSinceStartOfDay();
if(circleMetaData.count() > 0){
while(circleMetaData.at(circleSequence).frameNumber == frameSequence && circleSequence < circleCount-1)
{
if(circleMetaData.at(circleSequence).fillStatus)
{
p.setBrush(QBrush(QColor(circleMetaData.at(circleSequence).color)));
} else {
p.setPen(circleMetaData.at(circleSequence).color);
}
p.drawEllipse(circleMetaData.at(circleSequence).centerX,
circleMetaData.at(circleSequence).centerY,
circleMetaData.at(circleSequence).radiusX,
circleMetaData.at(circleSequence).radiusY);
circleSequence++;
}
}
qDebug()<<"6 "<<QTime::currentTime().msecsSinceStartOfDay();
p.end();
//imageEdited.save(QString("frame"+QString::number(frameSequence)+".jpg"));
frameSequence++;
cloneFrame.unmap();
}
if (surfaceFormat().pixelFormat() != frame.pixelFormat()
|| surfaceFormat().frameSize() != frame.size()) {
setError(IncorrectFormatError);
stop();
return false;
} else {
currentFrame = QVideoFrame(imageEdited);
// check by reconverting above to image if it is lost in this conversion
// else it is a rendering problem
widget->repaint(targetRect);
return true;
}
}
The Output of debugs in two different video case is:
30 FPS, 1920*1080 pixels
1 68862821
2 68862822
3 68862822
4 68862892
5 68862893
6 68862893
25 FPS, 1280*720 pixels
1 69319622
2 69319622
3 69319623
4 69319653
5 69319654
6 69319654
Please point me to some solution which is suitable for these two problems.
Above performance is captured with Windows 10 64 bit / Intel I5 2nd Gen 2.3 Gig 2 Cores / 8 gig RAM is someone wants to compare.