Orbbec and opencv Unity. Object on the floor detection

Question

I'm using "Opencv for unity" and an orbbec astra. I've the camera mounted on the ceiling and I need to detect the presence of objects on the floor. How can I remove the floor?

I can't use background subtraction because after some time, the objects will become background.

I tried to "record" a frame without objects as a background and every frame I subtract it from the depth image of the orbbec. It works very well but after 5-10 minutes it starts to make a lot of noise and it's not usable anymore.

if(recordBG){
   Imgcodecs.imwrite(Application.streamingAssetsPath + "/bg.jpg", imgMat); 
   recordBG = false;
   bg = imgMat;
}
imgMat = imgMat - bg;
Core.inRange(imgMat, scalarMin, scalarMax, imgMat);

I don't understand why after this time the depth image of the floor changes (the camera remains still).

Is there another way to remove the floor?

EDIT (Add images):

This is the Mat bg:

The "white" objects is the imgMat after the subtraction and I limit the detection at few cm from the floor (Core.inRange function)

This is the same thing after 5 minutes. This is my problem. nothing has moved between the 2 images

Answer 1

I have used a similar approach and I can tell you that depth cameras only provide an estimation of the depth each frame, so even when they are very accurate (this is not the case for Orbbec, you didn't mention what model are you using, but I had bad experience with Astra models) the values differs a little between frames (for the same points with stationary camera). What I do is to set a minimum height threshold, so when I subtract the background everything that is below this threshold is discarded. I use this to detect people and in my case is a reasonable option, in your case that depends on how tall your object are (I mean, are your object distinguishable from the depth noise ?). Also keep in mind that with this approach you can have negative values after the subtraction which can lead to weird behaviors when the data format for the depth is uint.

Answer 2

I haven't worked with depth cameras yet, but I have with normal cameras which return a 2d frame, one way you can solve this is that in the first few frames you are detecting the floor perfectly right ? create a mask of that segment and use it as a mask/fixed background in the other frames. This way you don't have to calculate/estimate the floor again and again.