Ideas for Efficient Slow Motion Video Playback in Raspberry Pi

Question

I'm new to Python and the Raspberry Pi, and I'm trying to write a program to record 8 sec video then play it back in slow motion. It doesn't have to be top quality video either, it just needs to run the program without a long delay.

A few years ago I witnessed a project that did exactly that. It recorded 8 sec of footage in response to a trigger, and took only a few seconds to process and play back that video in slow motion on the display. However, I have no idea how that person did it.

I've tried using ffmpeg but that takes over 90 seconds to create a slow motion video. I've tried OpenCV but to no avail. I've heard the MsgBox feature is an option too, but I can't find a full documentation for this feature. I'm open to using any file format, but currently the video outputs in H264. Does anyone have good ideas to create (or just playback) a slow motion video without a slow processing speed, please?

Here's the current code I use to record at 25 fps and play back in slow motion but it takes 1-2 min processing time (which I'd like to be only a few seconds):

import os
from picamera import PiCamera
from time import sleep
import vlc, subprocess

filename = "/home/pi/vid1.h264"
camera = PiCamera()

camera.start_preview()
camera.start_recording(filename)
sleep(8)
camera.stop_recording()
camera.stop_preview()

# Converts to slow motion video
input = filename
output = "/home/pi/slowvid1.h264"
command = f"ffmpeg -i {input} -vf setpts=2*PTS {output}"
os.system(command)

subprocess.Popen(['vlc',output, '--fullscreen','--play-and-exit'])

Here's a link to my issue with getting OpenCV to work Slow Motion Video Playback with OpenCV is not working (python)

Answer 1

Here's a way to do it. I use PiCamera to capture frames of MJPEG at 60 fps for 8 seconds. Read along section 4.7 in this document.

I chose the video port because it is the fastest capture method as it doesn't do sophisticated de-noising by averaging over multiple frames which slows things down.

I chose MJPEG "Motion JPEG" as the output format because that means the frames are JPEG-compressed on the GPU before I receive them, which means I need fewer CPU resources and less RAM because the frames are compressed.

As each frame arrives, I do no other processing than simply appending it to a list of JPEG-compressed frames in RAM (memory).

When the recording is complete, I simply iterate through the list of frames in memory, converting them back from JPEG to Numpy arrays which is what OpenCV uses for images, and display them. As they were acquired at 60fps, that means a new frame was recorded every 16ms. So, if I play back with a 64ms delay, it should appear 4x slower.

#!/usr/bin/env python3

import numpy as np
import cv2
import time
import picamera

class SplitFrames(object):
    def __init__(self):
        self.frame_num = 0
        # List of all the JPEG-encoded frames we receive
        self.frames = []
        # Total memory used
        self.memory = 0

    def write(self, buf):
        if buf.startswith(b'\xff\xd8'):
            # Start of new frame
            l = len(buf)
            print(f'DEBUG: New frame of {l} bytes')
            self.frames.append(buf)
            self.frame_num += 1
            self.memory    += l
        else:
            print(f'ERROR: partial frame of {len(buf)} bytes received belonging to previous frame')

################################################################################
# Main program - alter these variables according to needs
################################################################################
targetFPS, duration = 60, 8
width, height = 1024, 768
print(f'Recording frames of {width}x{height} at {targetFPS} fps for {duration} seconds')

################################################################################
# Recording loop
################################################################################
with picamera.PiCamera(framerate=targetFPS) as camera:
    camera.resolution=(width,height)
    camera.start_preview()
    # Give the camera some warm-up time
    time.sleep(2)
    output = SplitFrames()
    start = time.time()
    camera.start_recording(output, format='mjpeg')
    camera.wait_recording(duration)
    camera.stop_recording()
    finish = time.time()

# Calculate a few statistics
fps = output.frame_num / (finish - start)
avg = int(output.memory/output.frame_num)
MB  = output.memory/(1024*1024)
print(f'Captured {output.frame_num} frames at {fps:.3f} fps, average bytes/frame={avg}, total RAM={MB:.3f} MB')

################################################################################
# Playback loop - grab frames from list and display with delay
################################################################################
for frame_num, frame in enumerate(output.frames):
    print(f'DEBUG: Playing back frame {frame_num}')
    im = cv2.imdecode(np.frombuffer(frame, dtype=np.uint8), cv2.IMREAD_COLOR)
    cv2.imshow('Slow Motion Replay', im)
    cv2.waitKey(64)

As it runs, the output looks like this:

Recording frames of 1024x768 at 60 fps for 8 seconds
DEBUG: New frame of 26661 bytes
DEBUG: New frame of 33335 bytes
DEBUG: New frame of 33558 bytes
DEBUG: New frame of 34146 bytes
...
...
DEBUG: New frame of 34208 bytes
DEBUG: New frame of 34408 bytes
DEBUG: New frame of 34356 bytes
DEBUG: New frame of 34248 bytes
Captured 480 frames at 59.763 fps, average bytes/frame=36938, total RAM=16.909 MB

As you can see it uses just 17MB of RAM for 8s of 60fps @ 1024x768 pixels, which is under 2% of 1GB RasPi 3's RAM and less than 0.5% on a 4GB RasPi 4, so you could potentially run for longer without running out of RAM.

It then plays back INSTANTLY at quarter speed. Here is a mini animated GIF of playback after recording an iPhone high-speed timer:

Note 1: You are not obliged to use OpenCV for playback, you can use any other package that can display JPEGs.

Note 2: You could resize the image, annotate it, or process it in some way prior to displaying it in the replay loop.

Note 3: Note that you could replace the cv2.imshow() in the replay loop with a call to OpenCV VideoWriter() and save the video to disk, rather than display it on-screen if you wanted to.

Note 4: You could add a quality parameter to camera.start_recording() to control image quality/size. Or you could change to raw RGB or YUV format.

Keywords: RasPi, Raspberry Pi, RaspberryPi, image processing, video recording, PiCamera, high-speed, high speed, 60 fps, slow motion, slow-motion, slo-mo, buffer, in-memory buffering, replay.