Warp images using motion maps generated by opticalFlowLKDoG (Matlab 2015A)

Question

This question is based on a modified Matlab code from the online documentation for the optical flow system objects in version 2015a as appears in opticalFlowLK class

clc; clearvars; close all;
inputVid = VideoReader('viptraffic.avi');
opticFlow = opticalFlowLKDoG('NumFrames',3);
inputVid.currentTime = 2;
k = 1;
while inputVid.currentTime<=2 + 1/inputVid.FrameRate
 frameRGB{k} = readFrame(inputVid);
 frameGray{k} = rgb2gray(frameRGB{k});    
 flow{k} = estimateFlow(opticFlow,frameGray{k});
 k = k+1;
end

By looking at flow{2}.Vx and flow{2}.Vy I get the motion maps U and V that describe the motion from frameGray{1} to frameGray{2}.

Iwant to use flow{2}.Vx and flow{2}.Vy directly on the data in frameGray{1} in order to warp frameGray{1} to appear visually similar to frameGray{2}.

I tried this code:

[x, y] = meshgrid(1:size(frameGray{1},2), 1:size(frameGray{1},1));
frameGray1Warped = interp2(double(frameGray{1}) , x-flow{2}.Vx , y-flow{2}.Vy);

But it doesn't seem to do much at all except ruin the image quality (but the objects don't display any real motion towards their locations in frameGray{2}.

I added 3 images showing the 2 original frames followed by frame 1 warped using the motion field to appear similar to frame 2:

It can be seen easily that frame 1 warped to 2 is essentially frame 1 with degraded quality but the cars haven't moved at all. That is - the location of the cars is the same: look at the car closest to the camera with respect to the road separation line near it; it's virtually the same in frame 1 and frame 1 warped to 2, but is quite different in frame 2.