how to use ffmpeg with mjpeg_vaapi encoder (with hardware acceleration)?

Question

When I use ffmpeg with mjpeg encoder without hardware acceleration, everything works fine. I use following command:

ffmpeg -nostdin -rtsp_transport udp_multicast -allowed_media_types 'video' -i 'rtsp://MYIP' -ss 00:00:0.00 -f image2pipe -q:v 24 -vf scale=-2:480,format=yuv420p -c:v mjpeg -vframes 1 pipe:1

But, when I try to use mjpeg with hardware acceleration, I always get error:

ffmpeg -nostdin -rtsp_transport udp_multicast -allowed_media_types 'video' -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 'rtsp://MYIP' -ss 00:00:0.00 -f image2pipe -q:v 24 -vf scale=-2:480,format=yuv420p -c:v mjpeg_vaapi -vframes 1 pipe:1
ffmpeg version N-91514-gc51e0cd Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --pkg-config-flags=--static
  libavutil      56. 18.102 / 56. 18.102
  libavcodec     58. 21.106 / 58. 21.106
  libavformat    58. 17.101 / 58. 17.101
  libavdevice    58.  4.101 / 58.  4.101
  libavfilter     7. 26.100 /  7. 26.100
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
libva info: VA-API version 0.39.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
Input #0, rtsp, from 'rtsp://MYIP':
  Metadata:
    title           : Session streamed with GStreamer
    comment         : rtsp-server
  Duration: N/A, start: 0.180600, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, progressive), 800x600, 50 fps, 50 tbr, 90k tbn, 100 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (mjpeg_vaapi))
Impossible to convert between the formats supported by the filter 'graph 0 input from stream 0:0' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
Conversion failed!

I tried with some additional options but it still keeps failing:

ffmpeg -nostdin -rtsp_transport udp_multicast -allowed_media_types 'video' -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i 'rtsp://MYIP' -ss 00:00:0.00 -f image2pipe -q:v 24 -vf "scale_vaapi=w=800:h=600" -b:v 700k -r 25 -vcodec mjpeg_vaapi -vframes 1 pipe:1                
ffmpeg version N-91514-gc51e0cd Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --pkg-config-flags=--static
  libavutil      56. 18.102 / 56. 18.102
  libavcodec     58. 21.106 / 58. 21.106
  libavformat    58. 17.101 / 58. 17.101
  libavdevice    58.  4.101 / 58.  4.101
  libavfilter     7. 26.100 /  7. 26.100
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
libva info: VA-API version 0.39.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
Input #0, rtsp, from 'rtsp://MYIP':
  Metadata:
    title           : Session streamed with GStreamer
    comment         : rtsp-server
  Duration: N/A, start: 0.179933, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, progressive), 800x600, 50 fps, 50 tbr, 90k tbn, 100 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (mjpeg_vaapi))
[mjpeg_vaapi @ 0x374cd80] Encoding entrypoint not found (12 / 7).
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

Can please anyone suggest right options to make it work?

Answer 1

so the problem was that my Intel CPU platform was old (Broadwell) and JPEG encoding is supported starting Braswell. Please see all supported codecs by platform on the page

After running the same command on the machine with Skylake CPU on it, it work from the first try :)

Answer 2

When you trying to leverage Intel's hardware encoding, you need to check TWO things

1. rather your particular intel support the encoder you want
2. rather your ffmpeg binary fetched from somewhere compiled with the hardware acceleration library or not.

To find out what kind of encoder your Intel support, you simply type:

vainfo|grep -i enc

it will list the encoder which your processor support. In my case, for example, it yields the following:

libva info: VA-API version 1.13.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_13
libva info: va_openDriver() returns 0
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileHEVCMain               : VAEntrypointEncSlice

in my case, my intel cpu does have mjpeg encoding support.

if you can't find anything resembles mjpeg, then, you can pretty much give up the idea of using intel's hardware acceleration to accomplish what you want on that machine.

The next thing you need is to find out rather your ffmpeg binary support Intel's vaapi at the first place. First is to type:

ffmpeg -hide_banner -hwaccels

if one of those line has "vaapi," then, it means the ffmpeg compiled with Intel's hardware acceleration library. If your vainfo command has what you want but the above command doesn't display vaapi, then, you need to get another copy of ffmpeg binary. either from a different source, or compile it on your own.

Next, is to list all the ffmpeg encoder that actually uses vaapi, and hoping one of them resembles mjpeg:

ffmpeg -hide_banner -encoders|grep -i vaapi

In my case, I do have it:

 V....D h264_vaapi           H.264/AVC (VAAPI) (codec h264)
 V....D hevc_vaapi           H.265/HEVC (VAAPI) (codec hevc)
 V....D mjpeg_vaapi          MJPEG (VAAPI) (codec mjpeg)
 V....D mpeg2_vaapi          MPEG-2 (VAAPI) (codec mpeg2video)
 V....D vp8_vaapi            VP8 (VAAPI) (codec vp8)
 V....D vp9_vaapi            VP9 (VAAPI) (codec vp9)

So, in my case, the encoder is there.

if both vainfo and ffmpeg -hwaccels yield what you need but this last step doesn't, then, you need to get another ffmpeg binary from somewhere.

good luck.