FFmpeg - MJPEG decoding gives inconsistent values

Question

I have a set of JPEG frames which I am muxing into an avi, which gives me a mjpeg video. This is the command I run on the console:

ffmpeg -y -start_number 0 -i %06d.JPEG -codec copy vid.avi

When I try to demux the video using ffmpeg C api, I get frames which are slightly different in values. Demuxing code looks something like this:

AVFormatContext* fmt_ctx = NULL;
AVCodecContext* cdc_ctx = NULL;
AVCodec* vid_cdc = NULL;
int ret;
unsigned int height, width;

....
// read_nframes is the number of frames to read
output_arr = new unsigned char [height * width * 3 * 
                                sizeof(unsigned char) * read_nframes];

avcodec_open2(cdc_ctx, vid_cdc, NULL);

int num_bytes;
uint8_t* buffer = NULL;
const AVPixelFormat out_format = AV_PIX_FMT_RGB24;

num_bytes = av_image_get_buffer_size(out_format, width, height, 1);
buffer = (uint8_t*)av_malloc(num_bytes * sizeof(uint8_t));

AVFrame* vid_frame = NULL;
vid_frame = av_frame_alloc();
AVFrame* conv_frame = NULL;
conv_frame = av_frame_alloc();

av_image_fill_arrays(conv_frame->data, conv_frame->linesize, buffer, 
                     out_format, width, height, 1);

struct SwsContext *sws_ctx = NULL;
sws_ctx = sws_getContext(width, height, cdc_ctx->pix_fmt,
                         width, height, out_format,
                         SWS_BILINEAR, NULL,NULL,NULL);

int frame_num = 0;
AVPacket vid_pckt;
while (av_read_frame(fmt_ctx, &vid_pckt) >=0) {
    ret = avcodec_send_packet(cdc_ctx, &vid_pckt);
    if (ret < 0)
        break;

    ret = avcodec_receive_frame(cdc_ctx, vid_frame);
    if (ret < 0 && ret != AVERROR(EAGAIN) && ret != AVERROR_EOF)
        break;
    if (ret >= 0) {
        // convert image from native format to planar GBR
        sws_scale(sws_ctx, vid_frame->data, 
                  vid_frame->linesize, 0, vid_frame->height, 
                  conv_frame->data, conv_frame->linesize);

        unsigned char* r_ptr = output_arr + 
            (height * width * sizeof(unsigned char) * 3 * frame_num);
        unsigned char* g_ptr = r_ptr + (height * width * sizeof(unsigned char));
        unsigned char* b_ptr = g_ptr + (height * width * sizeof(unsigned char));
        unsigned int pxl_i = 0;

        for (unsigned int r = 0; r < height; ++r) {
            uint8_t* avframe_r = conv_frame->data[0] + r*conv_frame->linesize[0];
            for (unsigned int c = 0; c < width; ++c) {
                r_ptr[pxl_i] = avframe_r[0];
                g_ptr[pxl_i]   = avframe_r[1];
                b_ptr[pxl_i]   = avframe_r[2];
                avframe_r += 3;
                ++pxl_i;
            }
        }

        ++frame_num;

        if (frame_num >= read_nframes)
            break;
    }
}

...

In my experience around two-thirds of the pixel values are different, each by +-1 (in a range of [0,255]). I am wondering is it due to some decoding scheme FFmpeg uses for reading JPEG frames? I tried encoding and decoding png frames, and it works perfectly fine. I am sure this is something to do with the libav decoding process because the MD5 values are consistent between the images and the video:

ffmpeg -i %06d.JPEG -f framemd5 -
ffmpeg -i vid.avi -f framemd5 -

In short my goal is to get the same pixel by pixel values for each JPEG frame as I would I have gotten if I was reading the JPEG images directly. Here is the stand-alone bitbucket code I used. It includes cmake files to build code, and a couple of jpeg frames with the converted avi file to test this problem. (give '--filetype png' to test the png decoding).