object segmentation using mean shift

Question

i have this image:

I am interested to do segmentation only in the objects that appear in the image so i did something like this

import numpy as np
import cv2
from sklearn.cluster import MeanShift, estimate_bandwidth
#from skimage.color import rgb2lab
#Loading original image
originImg = cv2.imread('test/2019_00254.jpg')

# Shape of original image
originShape = originImg.shape


# Converting image into array of dimension [nb of pixels in originImage, 3]
# based on r g b intensities
flatImg=np.reshape(originImg, [-1, 3])


# Estimate bandwidth for meanshift algorithm
bandwidth = estimate_bandwidth(flatImg, quantile=0.1, n_samples=100)
ms = MeanShift(bandwidth = bandwidth, bin_seeding=True)

# Performing meanshift on flatImg
ms.fit(flatImg)

# (r,g,b) vectors corresponding to the different clusters after meanshift
labels=ms.labels_

# Remaining colors after meanshift
cluster_centers = ms.cluster_centers_

# Finding and diplaying the number of clusters
labels_unique = np.unique(labels)
n_clusters_ = len(labels_unique)
print("number of estimated clusters : %d" % n_clusters_)

segmentedImg = cluster_centers[np.reshape(labels, originShape[:2])]
cv2.imshow('Image',segmentedImg.astype(np.uint8))
cv2.waitKey(0)
cv2.destroyAllWindows()

but the problem is its doing segmentation in the whole image including the background so how can i do segmentation on the objects only note that i have bboxes coordinates of each object

Answer 1

I'd suggest you use a more straightforward input to understand (and feel) all the limitations behind the approach. The input you have is complex in terms of resolution, colors, scene complexity, object complexity, etc.

Anyway, to make this answer useful, let's do some experiments:

Detectron2, PointRend segmentation

Just in case you expect a complex model to handle the scene properly. Segmentation:

Masks:

No miracle here. The scene and objects are complex.

Monocular depth estimation

Let's try depth estimation as an obvious way to get rid of the background.

Depth (also check this example):

Result:

Part of the background is gone, but nothing to do with other objects. Long story short, start with something simple to see the exact way your solution works.

BTW, it is always hard to work with thin and delicate details, so it is better to avoid that complexity if possible.