Detect if a checkbox is checked OpenCV

Question

I am using OpenCV to process an image of a form which contains main checkboxes. I have used the pixel coordinates to crop around the checkbox so all I am left with is just the single checkbox. From there, I need to detect if there is anything inside of that checkbox (x, checkmark, slash, etc.) or not.

I used the code from this response and it was able to successfully draw a line around the checkbox, successfully detecting the checkbox (and part of the x). However, how do I then get it to tell me if the box is checked off in some way or if it's empty?

The checkbox in question before the code:

The code:

# Code courtesy of Sreekiran A R on Stack Overflow: https://stackoverflow.com/questions/63084676/checkbox-detection-opencv
image=box_5a_f

### converting BGR to Grayscale
gray_scale=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)

### Binarising image
th1,img_bin = cv2.threshold(gray_scale,180,225,cv2.THRESH_OTSU)

### defining kernels
lWidth = 2
lineMinWidth = 15

kernal1 = np.ones((lWidth,lWidth), np.uint8)
kernal1h = np.ones((1,lWidth), np.uint8)
kernal1v = np.ones((lWidth,1), np.uint8)

kernal6 = np.ones((lineMinWidth,lineMinWidth), np.uint8)
kernal6h = np.ones((1,lineMinWidth), np.uint8)
kernal6v = np.ones((lineMinWidth,1), np.uint8)

### finding horizontal lines
img_bin_h = cv2.morphologyEx(~img_bin, cv2.MORPH_CLOSE, kernal1h) # bridge small gap in horizonntal lines
img_bin_h = cv2.morphologyEx(img_bin_h, cv2.MORPH_OPEN, kernal6h) # kep ony horiz lines by eroding everything else in hor direction

## detect vert lines
img_bin_v = cv2.morphologyEx(~img_bin, cv2.MORPH_CLOSE, kernal1v)  # bridge small gap in vert lines
img_bin_v = cv2.morphologyEx(img_bin_v, cv2.MORPH_OPEN, kernal6v)# kep ony vert lines by eroding everything else in vert direction

def fix(img):
    img[img>127]=255
    img[img<127]=0
    return img
img_bin_final = fix(fix(img_bin_h)|fix(img_bin_v))

### getting labels
ret, labels, stats,centroids = cv2.connectedComponentsWithStats(~img_bin_final, connectivity=8, ltype=cv2.CV_32S)

### drawing recangles for visualisation
for x,y,w,h,area in stats[2:]:
    cv2.rectangle(image,(x,y),(x+w,y+h),(0,0,255),2)

The checkbox after the code:

I could also adjust my pixel coordinates to only crop the inside of the checkbox, but I'm still not sure what step to take from there and if it would really do anything to solve my issue.

Answer 1

I was having a bit of fun with this question, so my answer might not be the best. I thought that something recursive might work quite nicely here, something like find largest contour, get a smaller window, condition? If yes then break, if not then largest contour, get a smaller window, condition? and so on... until the condition is met. For my code I put the condition as the cv2.contourArea. Feel free to change, here's the code:

im = cv2.imread("x.png") # read the image
imCropped = im.copy()
imGray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY) # convert to gray
IterContour = 0 # just for displaying
while True:
    fig.clf()
    print("Iter "+str(IterContour)+"________________________________________________") # summary
    _, thresh = cv2.threshold(imGray, 127, 255, cv2.THRESH_BINARY_INV) # thresholding
    contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) # find contours
    contours = [contour for contour in contours if cv2.contourArea(contour) > 0] # remove contours with 0 area
    if len(contours) == 0:
        print("No contours found...")
        break
    else:
        print("Contours found...")
        ROI = max(contours, key=cv2.contourArea) # get largest contour
    if cv2.contourArea(ROI) < 500: # break if biggest contour has less than 500 pixels, OR any other condition!!!
        print("Contour too small, break...")
        break
    print("ROI has an area of "+str(cv2.contourArea(ROI))+" pixels") # summary
    x, y, w, h = cv2.boundingRect(ROI) # get bounding rect
    imCropped = imCropped[y+5:y+h-5, x+5:x+w-5] # crop the image
    imGray = cv2.cvtColor(imCropped, cv2.COLOR_BGR2GRAY) # convert to gray for next iter
    IterContour += 1 # increment
    plt.imshow(imGray) # plotting
    plt.title("Iter" + str(IterContour))
    plt.axis("off")
    fig.canvas.draw()
    time.sleep(1)
    cv2.imwrite(str(IterContour)+".png", imCropped)
plt.imshow(imCropped)
plt.axis("off")
plt.show()

The summary:

Iter 0________________________________________________
Contours found...
ROI has an area of 5258.5 pixels
Iter 1________________________________________________
Contours found...
ROI has an area of 582.5 pixels
Iter 2________________________________________________
Contours found...
Contour too small, break...

And here are the images, getting smaller each time. From here, you can use the window as you wish.

Hope this helps you further!

Answer 2

it checks the density of non-white pixels within each connected component (checkbox region) and determines whether the checkbox is likely checked or unchecked. The rectangles are drawn with green for checked checkboxes and red for unchecked checkboxes. Adjust the threshold_density value as needed for your specific scenario. hope it helps.

    # Iterate through connected components
for x, y, w, h, area in stats[2:]:        
    checkbox_roi = image[y:y+h, x:x+w]
            
    checkbox_gray = cv2.cvtColor(checkbox_roi, cv2.COLOR_BGR2GRAY)
            
    _, checkbox_bin = cv2.threshold(checkbox_gray, 180, 225, cv2.THRESH_OTSU)
            
    density = np.count_nonzero(checkbox_bin == 0) / (w * h)
    
    threshold_density = 0.5  # Adjust this threshold as needed
            
    if density > threshold_density:
        cv2.rectangle(image, (x, y), (x+w, y+h), (0, 255, 0), 2)  # Green for checked
    else:
        cv2.rectangle(image, (x, y), (x+w, y+h), (0, 0, 255), 2)  # Red for unchecked