I am working on a shape recognition app. At this moment a set of points (x,y) is determined by corner detector (red points, img. 2.). Four of these points (in red frames, img. 2.) are vertices of a rectangle (sometimes a little deformed rectangle). What would be the best way to find them among others?
Here is an example of an input image:

And it looks like this after corner detection:

This is not an answer to your question - it's just suggestion.
In my opinion corner detector is a bad way to detect rectangles - it will take much time to calculate all point distances as mathematician1975 suggested. You have to use another technique in this situation:
Update:
Here's another solution that should also work in gray images.
I take this
constantas 0.9.And this algorithm gave me next result:
Here's OpenCV code: