AWS Rekognition InvalidImageFormatException when jpeg source is mjpeg stream

Question

I'm pulling jpg frames out of a mjpg stream. These are valid jpg files and work in any image tool I've tried; however, Rekognition will not accept them either when sending it as Bytes, or when I move them to S3 and try that route.

I've made a few versions (attached here), all from the same source jpg (I would include them inline but I don't want image optimization code to alter them)

test.jpg - original frame

test-photoshop.jpg - opened in Photoshop, "save for web"d

test-imageoptim.jpg - run through ImageOptim (which I believe compresses with jpegtran)

Looking at these in a hex editor, the only difference I can't see is more exif data (using exiftool). When I run exiftool on the original, it still reports back all the basic details of the frame.

I'm assuming this is a bug with Rekognition, or there is some specific exif bit it's looking for that my mjpeg stream extraction is omitting. Maybe someone has information on why pulling jpeg frames from mjpeg isn't possible by just attaching the right start and end frame bytes.

Answer 1

From comment:

Just a guess, try chopping off the last 6 bytes, after the final FF D9 marker, perhaps those bytes are confusing Amazon.

Here's a snap from a hex viewer:

Answer 2

I can only make an educated guess because I do not have access to the application that is causing the problem. However, my guess is that it does not recognize the file format (or lack thereof) of the JPEG stream.

You have a Start of Frame Marker right after the Start of Image marker. That is legal according to the JPEG standard. However, there is nothing to indicate a file format. This is not strictly necessary to interpret the JPEG stream (assuming a 3 component scan is YCbCr encoded) but your application may be looking for some format specific marker (e.g., JFIF, EXIF).

Answer 3

Your images are fine. Your code is buggy or your permissions are not right. I used https://github.com/yourtion/Rekognition_API_NODEJS_SDK with your test.jpg and here is the result:

{
   "url":"POST",
   "face_detection":[
      {
         "boundingbox":{
            "tl":{
               "x":470.77,
               "y":109.23
            },
            "size":{
               "width":53.85,
               "height":53.85
            }
         },
         "confidence":0.25,
         "eye_left":{
            "x":489.3,
            "y":132.4
         },
         "eye_right":{
            "x":498,
            "y":128.9
         },
         "nose":{
            "x":504.1,
            "y":142.5
         },
         "mouth l":{
            "x":497.7,
            "y":152.1
         },
         "mouth_l":{
            "x":497.7,
            "y":152.1
         },
         "mouth r":{
            "x":502.4,
            "y":151.9
         },
         "mouth_r":{
            "x":502.4,
            "y":151.9
         },
         "pose":{
            "roll":-27.01,
            "yaw":86.01,
            "pitch":-12.56
         },
         "quality":{
            "brn":0.22,
            "shn":0.1
         },
         "emotion":{
            "happy":0.25,
            "surprised":0.09,
            "sad":0.07
         },
         "age":30.11,
         "smile":0.71,
         "glasses":0.79,
         "sunglasses":0.87,
         "beard":0.96,
         "mustache":0.52,
         "eye_closed":1,
         "mouth_open_wide":0.04,
         "beauty":0.5041,
         "sex":1
      }
   ],
   "ori_img_size":{
      "width":640,
      "height":480
   },
   "usage":{
      "quota":98,
      "status":"Succeed.",
      "api_id":"XXXXXXXXXXXXXXXXXXXX"
   }
}