What is a threshold in a Precision-Recall curve?

Question

I am aware of the concept of Precision as well as the concept of Recall. But I am finding it very hard to understand the idea of a 'threshold' which makes any P-R curve possible.

Imagine I have a model to build that predicts the re-occurrence (yes or no) of cancer in patients using some decent classification algorithm on relevant features. I split my data for training and testing. Lets say I trained the model using the train data and got my Precision and Recall metrics using the test data.

But HOW can I draw a P-R curve now? On what basis? I just have two values, one precision and one recall. I read that its the 'Threshold' that allows you to get several precision-recall pairs. But what is that threshold? I am still a beginner and I am unable to comprehend the very concept of the threshold.

I see in so many classification model comparisons like the one below. But how do they get those many pairs?

Model Comparison Using Precision-Recall Curve

Answer 1

ROC Curves:

• x-axis: False Positive Rate FPR = FP /(FP + TN) = FP / N
• y-axis: True Positive Rate TPR = Recall = TP /(TP + FN) = TP / P

Precision-Recall Curves:

• x-axis: Recall = TP / (TP + FN) = TP / P = TPR
• y-axis: Precision = TP / (TP + FP) = TP / PP

Your cancer detection example is a binary classification problem. Your predictions are based on a probability. The probability of (not) having cancer.

In general, an instance would be classified as A, if P(A) > 0.5 (your threshold value). For this value, you get your Recall-Precision pair based on the True Positives, True Negatives, False Positives and False Negatives.

Now, as you change your 0.5 threshold, you get a different result (different pair). You can already classify a patient as 'has cancer' for P(A) > 0.3. This will decrease Precision and increase Recall. You would rather tell someone that he has cancer even though he has not, to make sure that patients with cancer are sure to get the treatment they need. This represents the intuitive trade-off between TPR and FPR or Precision and Recall or Sensitivity and Specificity.

Let's add these terms as you see them more often common in biostatistics.

• Sensitivity = TP / P = Recall = TPR
• Specificity = TN / N = (1 – FPR)

ROC-curves and Precision-Recall curves visualize all these possible thresholds of your classifier.

You should consider these metrics, if accuracy alone is not a suitable quality measure. Classifying all patients as 'does not have cancer' will give you the highest accuracy but the values of your ROC and Precision-Recall curves will be 1s and 0s.

Answer 2

Adding to the answers, if you are plotting the curve, you can first get the probability by using predict_proba(X)

e.g.

>> clf = RandomForestClassifier(max_depth=2, random_state=0)
>> y_pred_prob = clf.predict_proba(X_test)

scikit-learn.org's reference on predict_proba()

then proceed to plot the curve with

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

precision, recall, thresholds = precision_recall_curve(y_test, y_pred_proba[:,1])

plt.plot(thresholds, precision[:-1], label='Precision')
plt.plot(thresholds, recall[:-1], label='Recall')
plt.xlabel('Threshold')
plt.ylabel('Score')
plt.legend()
plt.title('Precision-Recall vs. Threshold Curve')
plt.grid(True)
plt.show()

Answer 3

In addition to plotting, you can get an optimal threshold from the graphs using the below function:

from sklearn.metrics import precision_recall_curve
import numpy as np
   
def optimal_threshold_precision_recall_curve(gt, pmap):
        """Function to return an optimal thresholding value of the image
        """
        gt = gt.flatten()
        pmap = pmap.flatten()
        precision, recall, thresholds = precision_recall_curve(gt,pmap, pos_label=1)
        optimal_thresholds = sorted(list(zip(np.abs(precision - recall), thresholds)), key=lambda i: i[0], reverse=False)[0][1]
        optimal_mask = np.where(pmap>optimal_thresholds,1,0)
        return optimal_thresholds, optimal_mask

Note: The function takes the ground truth(gt) results together with the predicted probability maps (pmaps). I have flattened the inputs since the function accepts only 1D arrays. More details on the function and explanation can be found in this link.