Matlab VLFEAT -multiple matching

Question

I am performing SIFT matching with VLFEAT in Matlab.

A single match is simple to display: I followed the tutorial.

Update 1: (extracting the problem from my needs)

Next, I consider 4 different views of the scene: I want to match the feature found in the first camera (bottom left) with the others.

Images are already undistorted.

I could match the third image: I managed to correct the coordinates with offsets for a proper display. I set high threshold (fewer points) to have a more understandable image. My code is posted below, then there will be the question

Pointing out that (it does not affect the question or the answer, only the variable names in my code)

Since my 4 cameras are in fact a stereo camera moving in the space, the 4 cameras (and relative outputs) are:

Bottom left: left camera named a. The features in this image are fa, descriptors da...
Bottom right: left camera named a. The features in this image are fb, descriptors db...
Top left: left camera named a in the previous instant. The features in this image are fa_old, descriptors da_old...
Top right: right camera named b in the previous instant. The features in this image are fb_old, descriptors db_old...

Movements are smaller so I expected that SIFT could retrieve the same points.

This code finds points and performs the "blue matching" and "red matching"

%classic instruction for searching feature
    [fa,da] = vl_sift((Ia_f),'NormThresh', thresh_N, 'EdgeThresh', thresh_E) ;
% with the same line I obtain
%fa are features in the current left image (da are descriptors)
%fb are features in the current right image (db... )
%fa_old are features in the previous left image
%fb_old are features in the previous right image

%code from tutorials (find the feature)
[matches, scores] = vl_ubcmatch(da,db,thresh_SIFT) ;
[drop, perm] = sort(scores, 'descend') ;
matches = matches(:, perm);

%my code

figure(1) ; %clf ;
axis equal;
%prepare the image
imshow(cat(1,(cat(2, Ia_v_old, Ib_v_old)),cat(2,Ia_v,Ib_v)));

%matching between the left frames (current and previous)
[matches_prev, scores_prev] = vl_ubcmatch(da,da_old,thresh_SIFT) ; 
[drop_prev, perm_prev] = sort(scores_prev, 'descend') ;
matches_prev = matches_prev(:, perm_prev) ;

%find index of descriptors in common, write them in order
I = intersect(matches(1,:), matches_prev(1,:),'stable');
MI_1 = arrayfun(@(x)find(matches(1,:)==x,1),I);
MI_2 = arrayfun(@(x)find(matches_prev(1,:)==x,1),I);
matches_M = matches(:,MI_1(:));
matches_prev_M = matches_prev(:,MI_2(:));

%features coordinates in the current images (bottom)
xa = fa(1,matches_M(1,:)) + offset_column ;
xb = fb(1,matches_M(2,:)) + size(Ia,2); %+offset_column-offset_column ;
ya = fa(2,matches_M(1,:)) + offset_row + size(Ia,1);
yb = fb(2,matches_M(2,:)) + offset_row + size(Ia,1);            

%matching "in space" (blue lines)
space_corr = line([xa ; xb], [ya ; yb]) ;
set(space_corr,'linewidth', 1, 'color', 'b') ;
%plotting features
fa(1,:) = fa(1,:) + offset_column ;
fa(2,:) = fa(2,:) + offset_row + size(Ia,1);
vl_plotframe(fa(:,matches_M(1,:))) ;
fb(1,:) = fb(1,:) + size(Ia,2) ;
fb(2,:) = fb(2,:) + offset_row + size(Ia,1);
vl_plotframe(fb(:,matches_M(2,:))) ;

%matching "in time" %corrx and coor y are corrected offsets
xa2 = fa_old(1,matches_prev_M(2,:)) + corrx;  %coordinate per display
ya2 = fa_old(2,matches_prev_M(2,:)) - size(Ia,1) + corry;

fa_old(1,:) = fa_old(1,:) + corrx;
fa_old(2,:) = fa_old(2,:) - size(Ia,1) + corry;
fb_old(1,:) = fb_old(1,:) + corrx ;
fb_old(2,:) = fb_old(2,:) - size(Ia,1) + corry;

%plot red lines
time_corr = line([xa ; xa2], [ya ; ya2]) ;
set(time_corr,'linewidth', 1, 'color', 'r') ;
%plot feature in top left image
vl_plotframe(fa_old(:,matches_prev_M(2,:))) ;
%plot feature in top right image
vl_plotframe(fb_old(:,matches_ex_M(2,:))) ;

All works. I thought to repeat few lines of code and find the proper matches_ex_M index array in the proper order and finally connect the feature in the last (top right) image (with any one of the other images)

% one of many tries (all wrong)
[matches_ex, scores_ex] = vl_ubcmatch(da_old,db_old,thresh_SIFT) ;
[drop_ex, perm_ex] = sort(scores_ex, 'descend') ;
matches_ex = matches_ex(:, perm_ex);
Ib = intersect(matches_prev_M(2,:), matches_ex(1,:),'stable');
MIb_2 = arrayfun(@(x)find(matches_ex(1,:)==x,1),Ib);
matches_ex_M = matches_ex(:,MIb_2(:));

The problem is that a new intersection will cause a new reorder, and all the matches will be wrong. But I have admitted I have no more ideas, after trying all possible combinations of matching index arrays. The problem is that I can't perform either intersection between 3 arrays simultaneously, nor changing their order. Featured are well displayed in all 4 images and I can perform single matches from any image to another in different scripts. In the top right images, there are the same features but with different order.

What I obtain (obviously wrong)

Synthesizing my problem:

I thought I should change the order of the points in the top right frame to have a good "yellow" matching, but I don't know how to make it without changing the order in the top left (this will destroy the "red" matching" and/or the "blue matching")

Any idea? Any different strategies? Thank you all in advance.

UPDATE 2: After thinking to switch from MATLAB + VLFEAT to Python(2.7) + OpenCV (2.4.13) (I'd prefer to have a solution in Matlab and VLFEAT) I found this answer. Someone could do that in C++. But I'm unable to convert it nor in Matlab neither in Python. A pythonic solution could be accepted as well (added proper tags for that reason).