Evaluating a classifier: confusion matrix, precision and recall

A binary classifier gets it wrong in two ways: it calls something negative a positive, or it lets a positive slip through. The confusion matrix counts the four cases; the metrics come out of it.

The confusion matrix

Four counts (1 = positive):

• TP (true positive): it was 1, predicted 1.
• FN (false negative): it was 1, predicted 0.
• FP (false positive): it was 0, predicted 1.
• TN (true negative): it was 0, predicted 0.

Precision, recall and F1

Precision answers "of the ones I flagged as positive, how many really were?": TP / (TP + FP). Recall answers "of the positives that exist, how many did I find?": TP / (TP + FN). F1 is the harmonic mean of the two, and it is low if either one is low.

actual = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
pred = [1, 1, 1, 0, 1, 1, 0, 0, 0, 0]
tp = sum(a == 1 and p == 1 for a, p in zip(actual, pred))
fn = sum(a == 1 and p == 0 for a, p in zip(actual, pred))
fp = sum(a == 0 and p == 1 for a, p in zip(actual, pred))
tn = sum(a == 0 and p == 0 for a, p in zip(actual, pred))
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print("TP FN FP TN:", tp, fn, fp, tn)
print(f"precision={precision:.2f} recall={recall:.2f} F1={f1:.2f}")

TP FN FP TN: 3 1 2 4
precision=0.60 recall=0.75 F1=0.67

The choice depends on the cost of the error: in fraud, FN weighs more (recall); when blocking customers, FP does (precision).

Pitfall

With imbalanced classes, accuracy is misleading. A model that always answers "negative" is right 98% of the time on a dataset with 2% positives and detects none of them:

actual = [1] * 2 + [0] * 98
pred = [0] * 100
hits = sum(a == p for a, p in zip(actual, pred))
tp = sum(a == 1 and p == 1 for a, p in zip(actual, pred))
print("accuracy:", hits / len(actual))
print("recall:", tp / sum(actual))

accuracy: 0.98
recall: 0.0

Report precision and recall for the rare class. If TP + FP is zero, precision is undefined: handle that case.