Clustering similar lines with HDBSCAN

Question

The image above is a frame from a video. The ultimate goal is to detect the gate. What I want to do is cluster lines similarly to the circles, where the lines that are not circled are outliers. My findings tells me this is a HDBSCAN problem so I have attempted to implement HDBSCAN like this:

import cv2
import numpy as np
import hdbscan

gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# Create default Fast Line Detector (FSD)
fld = cv2.ximgproc.createFastLineDetector(_length_threshold=60,
                                          _canny_th1=5,
                                          _canny_th2=5,
                                          _canny_aperture_size=3,
                                          _do_merge=True
                                          )

# Detect lines in the image
lines = fld.detect(gray_img)

cluster_lines = []
if type(lines) is not type(None):
    for line in lines:
        x1, y1, x2, y2 = line[0]
        cluster_lines.append([(x1 + x2) // 2, (y1 + y2) // 2])
        # cluster_lines.append([x1, y1, x2, y2])

line_clusterer = hdbscan.HDBSCAN(min_cluster_size=3,
                                 cluster_selection_epsilon=0.5)
line_clusterer.fit(cluster_lines)
labels = line_clusterer.labels_
cluster_max = line_clusterer.labels_.max()

I have tried 2 approaches, one is to just pass in a list of the lines as they are, which gave really bad results and Im assuming that is because the HDBSCAN doesnt know the coordinates represent a line. The second is to pass in a list of the midpoint of every line, which works better but still not great. Is there a way that I can improve this and get something more similar to the desired result?

Original Image