PyAV: how to display multiple video streams to the screen at the same time

Question

I'm just learning to work with video frames and new to python language. I need to display multiple video streams to the screen at the same time using PyAV.

The code below works fine for one camera. Please help me to display multiple cameras on the screen. What should I add or fix in this code?

dicOption={'buffer_size':'1024000','rtsp_transport':'tcp','stimeout':'20000000','max_delay':'200000'}
video = av.open("rtsp://viewer:[email protected]:80/1", 'r',format=None,options=dicOption, metadata_errors='nostrict')
try:
    for packet in video.demux():
        for frame in packet.decode():
            if packet.stream.type == 'video':
            print(packet)
            print(frame)
            img = frame.to_ndarray(format='bgr24')
            #time.sleep(1)
            cv2.imshow("Video", img)
       if cv2.waitKey(1) & 0xFF == ord('q'):
           break
except KeyboardInterrupt:
    pass
cv2.destroyAllWindows()

Answer 1

Playing multiple streams with PyAV is possible but not trivial. The main challenge is decoding multiple streams simultaneously, which in a single-threaded program can take longer than the frame rate of the videos would require. Unfortunately threads won't be of help here (Python allows only one thread to be active at any given time), so the solution is to build a multi-process architecture.

I created the code below for a side project, it implements a simple multi-stream video player using PyAV and OpenCV. It creates a separate background process to decode each stream, using queues to send the frames to the main process. Because the queues have limited size, there is no risk of decoders outpacing the main process — if a frame is not retrieved by the time the next one is ready, its process will block until the main process catches up.

All streams are assumed to run at the same frame rate.

import av
import cv2
import numpy as np

import logging
from argparse import ArgumentParser
from math import ceil
from multiprocessing import Process, Queue
from time import time


def parseArguments():
    r'''Parse command-line arguments.
    '''
    parser = ArgumentParser(description='Video player that can reproduce multiple files simultaneoulsy')
    parser.add_argument('paths', nargs='+', help='Paths to the video files to be played')
    parser.add_argument('--resolution', type=int, nargs=2, default=[1920, 1080], help='Resolution of the combined video')
    parser.add_argument('--fps', type=int, default=15, help='Frame rate used when playing video contents')

    return parser.parse_args()


def decode(path, width, height, queue):
    r'''Decode a video and return its frames through a process queue.

        Frames are resized to `(width, height)` before returning.
    '''
    container = av.open(path)
    for frame in container.decode(video=0):
        # TODO: Keep image ratio when resizing.
        image = frame.to_rgb(width=width, height=height).to_ndarray()
        queue.put(image)

    queue.put(None)


class GridViewer(object):
    r'''Interface for displaung video frames in a grid pattern.
    '''
    def __init__(self, args):
        r'''Create a new grid viewer.
        '''
        size = float(len(args.paths))
        self.cols = ceil(size ** 0.5)
        self.rows = ceil(size / self.cols)

        (width, height) = args.resolution
        self.shape = (height, width, 3)

        self.cell_width = width // self.cols
        self.cell_height = height // self.rows

        cv2.namedWindow('Video', cv2.WINDOW_NORMAL | cv2.WINDOW_KEEPRATIO | cv2.WINDOW_GUI_EXPANDED)
        cv2.resizeWindow('Video', width, height)

    def update(self, queues):
        r'''Query the frame queues and update the viewer.

            Return whether all decoders are still active.
        '''
        grid = np.zeros(self.shape, dtype=np.uint8)
        for (k, queue) in enumerate(queues):
            image = queue.get()
            if image is None:
                return False

            (i, j) = (k // self.cols, k % self.cols)
            (m, n) = image.shape[:2]

            a = i * self.cell_height
            b = a + m

            c = j * self.cell_width
            d = c + n

            grid[a:b, c:d] = image

        grid = cv2.cvtColor(grid, cv2.COLOR_RGB2BGR)
        cv2.imshow('Video', grid)
        cv2.waitKey(1)

        return True


def play(args):
    r'''Play multiple video files in a grid interface.
    '''
    grid = GridViewer(args)

    queues = []
    processes = []
    for path in args.paths:
        queues.append(Queue(1))
        processes.append(Process(target=decode, args=(path, grid.cell_width, grid.cell_height, queues[-1]), daemon=True))
        processes[-1].start()

    period = 1.0 / args.fps
    t_start = time()
    t_frame = 0

    while grid.update(queues):
        # Spin-lock the thread as necessary to maintain the frame rate.
        while t_frame > time() - t_start:
            pass

        t_frame += period

    # Terminate any lingering processes, just in case.
    for process in processes:
        process.terminate()


def main():
    logging.disable(logging.WARNING)

    play(parseArguments())


if __name__ == '__main__':
    main()