Can I have a rawvideo stream in a MPEG TS container

Question

I receive an MPEG TS container over network (UDP). It contains two streams: an mpeg2video vidoe stream with yuv420p pixel format and a data stream encoded using a propitiatory KLV format.

My receiver program must be in Python. So, I can't use FFMPEG library (like AVFormat, AVCodec) directly.

Now my problem is as follows:

I need to receive video frames and save them as RGB image as raw numpy array. I also need for each frame to parse the corresponding KLV data. There is a one to one relationship between video frames and KLV data units.

I thought I use ffprobe to output the packets including their payload data from incoming container and then parse the output of ffprobe to get the images and metadata:

$ ffprobe -show_packets -show_data -print_format json udp://127.0.0.1:12345 > test_video.packets.data.json 

This gives me an output (in test_video.packets.data.json file) like:

{
    "codec_type": "video",
    "stream_index": 0,
    "pts": 140400,
    "pts_time": "1.560000",
    "dts": 136800,
    "dts_time": "1.520000",
    "duration": 3600,
    "duration_time": "0.040000",
    "size": "21301",
    "pos": "3788012",
    "flags": "K_",
    "side_data_list": [
        {
            "side_data_type": "MPEGTS Stream ID"
        }
    ],
    "data": "... "
},
{
    "codec_type": "data",
    "stream_index": 1,
    "pts": 140400,
    "pts_time": "1.560000",
    "dts": 140400,
    "dts_time": "1.560000",
    "size": "850",
    "pos": "3817904",
    "flags": "K_",
    "side_data_list": [
        {
            "side_data_type": "MPEGTS Stream ID"
        }
    ],
    "data": ".... "
}

I can extract the KLV data from the data packets and parse it. However the data from the video packets in encoded as mpeg2video video with yuv420p pixel format.

My Questions:

1. How can I get the raw pixel values from that mpeg2 encoded payload?
2. Is it possible to use ffmpeg to receive the original container and copy it (with both streams) into a new container, but with raw video instead of mpeg2 video? if yes, how? what should be the command? I tried for example: ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -codec rawvideo -pix_fmt rgb24 -map 0:1 -codec copy -f mpegts udp://127.0.0.1:11112, but it gives me again mpeg2 encoded video data in payload of video packets

Answer 1

MPEG-TS supports a limited number of video codecs. However, ffmpeg's muxer will silently mux even unsupported streams as private data streams.

To mux a raw RGB stream, convert to rgb24 pixel format and code using rawvideo codec.

ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -map 0:1 -c copy -c:v rawvideo -pix_fmt rgb24 -f mpegts udp://127.0.0.1:11112