Disparity Map obtained through StereoSGBM is flickering

Question

I am trying to implement stereo vision using 2 RPI v2 Cameras kept 8.43cm apart. After performing camera calibration and rectification, I used StereoSGBM to compute disparity map.

But for some reason, the disparity map obtained is flickering too much.

Here is how it looks. disparity map

This is the left and right rectified image: rectified image

I made sure I cover every corner while calibrating.

I've tried same pipeline ( Different Parameter Value ) in gazebo and it works really well.

Here is how I am initiating the block Matcher.

    wsize = 15
    max_disp = 16*3
    sigma = 1.5
    lmbda = 8000.0

    block_size = 5
    min_disparity = 0
    num_disparities = 16
    uniqueness_ratio = 10
    speckle_window_size = 200
    speckle_range = 32
    disp_12_max_diff = 1
    pre_filter_cap = 63
    P1 = 8 * 3 * block_size * block_size
    P2 = 32 * 3 * block_size * block_size
    mode = cv2.StereoSGBM_MODE_SGBM_3WAY

    left_matcher = cv2.StereoSGBM_create(minDisparity=min_disparity,
                            numDisparities=num_disparities,
                            blockSize=block_size,
                            P1=P1,
                            P2=P2,
                            disp12MaxDiff=disp_12_max_diff,
                            preFilterCap=pre_filter_cap,
                            uniquenessRatio=uniqueness_ratio,
                            speckleWindowSize=speckle_window_size,
                            speckleRange=speckle_range,
                            mode=mode)

    right_matcher = cv2.ximgproc.createRightMatcher(left_matcher)
    wls_filter = cv2.ximgproc.createDisparityWLSFilter(left_matcher)

` ret_val_l, left_image = left.read() ret_val_r, right_image = right.read()

        if ret_val_r is None and ret_val_l is None:
            break


        R1, R2, P1, P2, Q, ROI1, ROI2  = cv2.stereoRectify(camera_matrix_1, distortion_coefficients_1, camera_matrix_2, distortion_coefficients_2, left_image.shape[:2], R, T)

        
        left_image_rect = np.zeros_like(left_image)
        x, y, w, h = ROI1
        left_image_rect[y:y+h, x:x+w] = left_image[y:y+h, x:x+w]

        right_image_rect = np.zeros_like(right_image)
        x, y, w, h = ROI2
        right_image_rect[y:y+h, x:x+w] = right_image[y:y+h, x:x+w]

        left_image_gray = cv2.cvtColor(left_image_rect, cv2.COLOR_BGR2GRAY)
        right_image_gray = cv2.cvtColor(right_image_rect, cv2.COLOR_BGR2GRAY)

        left_disparity = left_matcher.compute(left_image_gray, right_image_gray)# .astype(np.float32)
        right_disparity = right_matcher.compute(right_image_gray, left_image_gray)# .astype(np.float32)

        wls_filter.setLambda(lmbda)
        wls_filter.setSigmaColor(sigma)  
        filtered_disp = wls_filter.filter(left_disparity, left_image_gray, disparity_map_right=right_disparity)

        jet_colormap = cv2.applyColorMap(filtered_disp.astype(np.uint8), cv2.COLORMAP_JET)`

I've tried with stereoBM, same issue. How do I make it consistent?