Confusion Matrix Calculator

Use a confusion matrix when evaluating a binary classifier—whether it's a machine learning model, diagnostic test, or any system that makes yes/no predictions. The matrix reveals not just overall accuracy but the specific types of errors being made.

What Is a Confusion Matrix?

A confusion matrix cross-tabulates predictions against actual outcomes:

The Four Outcomes

Key Metrics

Accuracy

$Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$

Overall proportion correct. Can be misleading with imbalanced classes.

Precision (Positive Predictive Value)

$Precision = \frac{TP}{TP + FP}$

Of all positive predictions, how many were correct? "When I say yes, how often am I right?"

Recall (Sensitivity, True Positive Rate)

$Recall = \frac{TP}{TP + FN}$

Of all actual positives, how many did I find? "Of all the yeses, how many did I catch?"

Specificity (True Negative Rate)

$Specificity = \frac{TN}{TN + FP}$

Of all actual negatives, how many did I correctly identify?

F1 Score

$F_1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}$

Harmonic mean of precision and recall. Balances both concerns.

Negative Predictive Value (NPV)

$NPV = \frac{TN}{TN + FN}$

Of all negative predictions, how many were correct?

Example Calculation

Spam filter results on 1000 emails:

• TP = 80, FP = 30, FN = 20, TN = 870
• Total = 1000

Calculations:

• Accuracy = (80 + 870) / 1000 = 95%
• Precision = 80 / 110 = 72.7%
• Recall = 80 / 100 = 80%
• Specificity = 870 / 900 = 96.7%
• F1 = 2 × (0.727 × 0.80) / (0.727 + 0.80) = 76.2%

The Precision-Recall Trade-off

You can't maximize both simultaneously. Lowering the classification threshold:

• ↑ Recall (catch more positives)
• ↓ Precision (more false alarms)

When to prioritize recall:

• Disease screening (don't miss cases)
• Fraud detection (investigate more)
• Safety-critical systems

When to prioritize precision:

• Spam filtering (minimize annoyance)
• Recommendations (quality over quantity)
• Costly interventions

Additional Metrics

False Positive Rate (Fall-out)

$FPR = \frac{FP}{FP + TN} = 1 - Specificity$

False Negative Rate (Miss Rate)

$FNR = \frac{FN}{TP + FN} = 1 - Recall$

False Discovery Rate

$FDR = \frac{FP}{TP + FP} = 1 - Precision$

Matthews Correlation Coefficient

$MCC = \frac{TP \times TN - FP \times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$

Ranges from -1 to +1. Balanced measure even with imbalanced classes.

Balanced Accuracy

$Balanced\ Accuracy = \frac{Sensitivity + Specificity}{2}$

Average of performance on each class.

The Class Imbalance Problem

With rare events (e.g., 1% positive rate), accuracy is misleading:

Example: 10,000 cases, 100 positive, 9,900 negative

• Classifier predicts "negative" for everything
• Accuracy = 99%!
• But recall = 0% (missed all positives)

Solutions:

• Use precision, recall, F1 instead of accuracy
• Use balanced accuracy or MCC
• Consider cost-weighted metrics
• Adjust classification threshold

ROC and AUC

The ROC curve plots recall vs. FPR at various thresholds:

• Area Under Curve (AUC) summarizes overall performance
• AUC = 0.5 = random guessing
• AUC = 1.0 = perfect classification

Precision-Recall Curve

For imbalanced data, precision-recall curves are more informative than ROC:

• Plot precision vs. recall at various thresholds
• Area under PR curve (PR-AUC) summarizes performance
• More sensitive to changes in minority class performance

Multi-Class Extension

For k classes, the confusion matrix is k × k:

Calculate metrics per-class, then average:

• Macro average: Average across classes (equal weight)
• Micro average: Aggregate then calculate (weight by class size)
• Weighted average: Weight by class frequency

Applications

Medical Diagnosis

• TP: Disease detected, patient has disease
• FP: Disease detected, patient healthy (unnecessary treatment)
• FN: Disease missed (dangerous!)
• TN: Healthy correctly identified

Fraud Detection

• TP: Fraud detected
• FP: False alarm (customer inconvenienced)
• FN: Fraud missed (financial loss)
• TN: Legitimate transaction approved

Search Engines

• TP: Relevant result shown
• FP: Irrelevant result shown
• FN: Relevant result not shown
• TN: Irrelevant result correctly excluded

Cost-Sensitive Classification

When errors have different costs:

$Total\ Cost = C_{FP} \times FP + C_{FN} \times FN$

Adjust threshold to minimize total cost, not maximize accuracy.

Example: If missing a cancer case costs 100× more than a false positive, weight FN heavily.

Threshold Selection

Most classifiers output probabilities. Choose threshold based on:

1. Business requirements: Required precision or recall level
2. Cost analysis: Minimize expected cost
3. F1 optimization: Maximize F1 score
4. Youden's J: Maximize (Sensitivity + Specificity - 1)

Confidence Intervals

For proportions like precision and recall, use binomial confidence intervals:

$\hat{p} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

Or use bootstrap for more complex metrics like F1.

Reporting Guidelines

Include:

• Full confusion matrix
• Precision, recall, F1 at chosen threshold
• ROC-AUC or PR-AUC for threshold-independent evaluation
• Class distribution in test set
• Confidence intervals for key metrics

Example: "The model achieved 95% accuracy on the test set (n=1000). However, given the 10:1 class imbalance, we focus on precision (72.7%) and recall (80.0%), yielding F1 = 76.2%. The ROC-AUC was 0.91."

Related Calculations

• ROC-AUC Calculator — Threshold-independent evaluation
• Sensitivity & Specificity — Diagnostic test metrics
• Phi Coefficient — Association measure for 2×2 tables
• Chi-Square Test — Test for association

Summary

The confusion matrix breaks down classifier performance into true/false positives/negatives. Accuracy alone is insufficient, especially with imbalanced classes. Use precision (quality of positive predictions), recall (completeness of positive detection), and F1 (balance of both). Consider the costs of different errors when choosing thresholds. MCC and balanced accuracy are robust to class imbalance. Always report the full matrix alongside summary metrics.