CamShift only tracks object sometime, even though BackProjection is very good

Question

edit: it seems like the side of the frame the object enters from matters. Not sure why it would, but entering from the right it always get picked up, from the top, it never does, until it gets close to the right side.

In short, I'm trying to track an object using OpenCV and CamShift. I have calculated and displayed my backprojection image and it looks fantastic. Its basically a white blob on a black background. However, my program only picks it up sometimes. The search window for CamShift encompasses the entire image, and then shrinks down to the object when it comes on screen, then expands out again when the object leaves. This works fine about 2/3 times. I don't understand what is wrong the third time.

https://i.stack.imgur.com/Epx28.jpg : Album of screen shots of showing what I mean. The green rectangle is the search window (trackWindow) and red is the rotatedRect output of CamShift (trackBox).

Any ideas? I tried playing with the criteria, but that didn't seem to do much.

calcBackProject(&image_hue, 1, 0, hist, backproj, &ranges, 1, true );
erode(backproj, backproj, Mat());
dilate(backproj, backproj, Mat());

backproj=backproj*10;
imshow ("", backproj);

cout << "width " << trackWindow.width << "\n";
cout << "height " << trackWindow.height << "\n";    

RotatedRect trackBox = CamShift(backproj, trackWindow, TermCriteria( CV_TERMCRIT_EPS | CV_TERMCRIT_ITER, 10, 1 ));

rectangle(image,trackWindow, Scalar(0,255,0),1,8,0);

Point2f vertices[4];
trackBox.points(vertices);
for (int i = 0; i < 4; i++)
{
line(image, vertices[i], vertices[(i+1)%4], Scalar(0,0,255));
}