Is there any way to capture screenshot from gstreamer stream by user demand?

Question

So my situation is I have a gstreamer pipeline (deepstream more precisely), that runs from C++ program, I have a function that checks user commands and I'm able to pause/resume pipeline that way. My question is there way to get screenshot from pipeline's videostream likewise?

Answer 1

If you are looking to check if the video is making it through your pipeline, use the "identity dump=1" element in your pipeline. This will display frames in a hexdump manner if the video is actually making through at the point at which this identity element is inserted.

Answer 2

The simplest answer would be to add a pad probe somewhere in your pipeline and, based on your flags either store or ignore the buffer. An example would be something like:

static GstPadProbeReturn 
pad_cb(GstPad *pad, GstPadProbeInfo *info, gpointer u_data){
    // for simplicity i am ignoring any safety checks here
    if(!SHOULD_SAVE_IMG) return GST_PAD_PROBE_OK;
    GstBuffer *buf = (GstBuffer *) info->data;
    GstMapInfo in_map;
    gst_buffer_map(buf, &map, GST_MAP_READ);
    
    unsigned char *buffer = (unsigned char *)in_map.info;
    //either use openCV to store image or if at this point the 
    //buffer is already encoded you could directly write this
    //to a file using standard C/C++ apis
    //...
    return GST_PAD_PROBE_OK;
}

Chosing the approach described above, take care where and in which format the buffer currently is, you might have to do some thinkering if using openCV to get the correct Mat type regarding its depth, this is especially true if you are running nvstreammux since the stream would be in NV12 format.

The other alternatives would be a little bit more complex regarding the setup. In particular you could expoit the valve element with its drop property combined with the tee element. In this way with a simple probe in the sink of the valve you could easily enable or disable the branch. An example pipeline would look like the following.

/*
src -> ... -> tee -> valve -> nvjpegenc -> multifilesink
                  -> (your pipeline to display the video)
*/
static GstPadProbeReturn
valve_sink_pad_probe(GstPad *pad, GstPadProbeInfo *info, gpointer u_data){
    gboolean is_drop;
    g_object_get(G_OBJECT(valve),
                "drop", &is_drop,
                NULL);
    //change only if different
    if(is_drop == SHOULD_SAVE_IMG){
        g_object_set(G_OBJECT(valve),
                    "drop",
                    !is_drop,
                    NULL);
        //flip value (only if is true) so 
        //that only the next time drop is
        //automatically set to true
        //achieving the "screenshot" effect
        if(SHOULD_SAVE_IMG) 
            SHOULD_SAVE_IMG = !SHOULD_SAVE_IMG;
    }
    return GST_PAD_PROBE_OK;
}

Using the fact that you are using Deepstream, in the above pipeline nvjpegenc expects the frame buffer to be in NV12 or I420 format therefore a nvstreammux or a nvvideoconverter would be required if the stream is not already in that format.