How to detect the exact boundary of a Sudoku using OpenCV when there are multiple external boundaries?

Question

I was trying to extract a sudoku grid using OpenCV by finding the square contour with the largest area. But some sudokus have more than one outer boundary, thus the program detects only the outermost boundary. This affects the later parts of the program because the sudoku is split into 9 parts row wise and column wise, to extract the digit from each cell. How can I modify the main_outline() function to detect the sudoku outline, which is the actual boundary of the sudoku and not the additional outer boundaries?

def main_outline(contour):
                biggest = np.array([])
                max_area = 0
                for i in contour:
                    area = cv2.contourArea(i)
                    if area >50:
                        peri = cv2.arcLength(i, True)
                        approx = cv2.approxPolyDP(i , 0.02* peri, True)
                        if area > max_area and len(approx) ==4:
                            biggest = approx
                            max_area = area
                return biggest ,max_area

su_contour_1= su_puzzle.copy()
su_contour_2= su_puzzle.copy()
su_contour, hierarchy = cv2.findContours(preprocess(su_puzzle),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
cv2.drawContours(su_contour_1, su_contour,-1,(0,255,0),3)
            
black_img = np.zeros((450,450,3), np.uint8)
su_biggest, su_maxArea = main_outline(su_contour)

The problem can be explained with these images:

Input image:

Extracted image with current program:

Expected extracted image:

I tried using a recursive function to detect the inner boundaries, but it failed since after detecting the exact boundary, it will further find the contours, which will be detected as the individual sudoku grids, leading to wrong results. I could not find a good stopping criteria.

Answer 1

If the boundary is always blue, this might work quite nicely:

im = cv2.imread("sudoku.png") # read im
b,g,r = cv2.split(im) # split to bgr
bThresh = cv2.threshold(b, 0, 1, cv2.THRESH_OTSU + cv2.THRESH_BINARY_INV)[1] # get the threshold
cnts, _ = cv2.findContours(bThresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE) # find the contours
largestCnt = max(cnts, key = cv2.contourArea) # find the largest contour
x, y, w, h = cv2.boundingRect(largestCnt) # get bounding rect
croppedIm = im[y:y+h, x:x+w] # crop
cv2.imwrite("Cropped.png", croppedIm) # save

Visualizing the following code:

imContoured = cv2.drawContours(cv2.merge((r,g,b)).copy(), largestCnt, -1, (255,0,0), 5) # draw largest contour with thick line, red
plt.figure()
plt.imshow(imContoured)
plt.axis("off")

give the following:

The final saved image: