Evaluation Metrics: Accuracy Is Not Enough
Accuracy is the first metric everyone learns and the first one to mislead you. If your data is 95% one class, a dumb model that always predicts that class gets 95% accuracy and is useless. You need metrics that reflect the cost of being wrong.
Precision answers: when you predicted positive, how often were you right? Recall answers: of all real positives, how many did you find? F1 balances the two. ROC shows tradeoffs across thresholds.
Choose the metric that matches your product. For spam, high precision matters. For disease screening, high recall matters. Reporting the wrong metric is how projects fail in production while looking good in notebooks.
Precision, recall, F1
Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = harmonic mean. Use classification_report to see all at once.
from sklearn.metrics import precision_score, recall_score, f1_score
print(precision_score(y_test, pred))
print(recall_score(y_test, pred))ROC and thresholds
ROC curve shows true positive rate vs false positive rate as you move the threshold. Use it to pick a threshold that fits your cost.
from sklearn.metrics import roc_curve
fpr, tpr, thr = roc_curve(y_test, proba)Syntax
from sklearn.metrics import classification_report
print(classification_report(y_test, pred, target_names=["ham","spam"]))The 95% accuracy trap
A model that always predicts the majority class gets 95% accuracy on 95/5 data but finds zero of the 5 you care about. Always check precision/recall for the minority class. This Kaggle discussion is the top upvoted post in “metrics”.
print(classification_report(y_test, pred, digits=3)) # check minority classGood to know
- Accuracy is misleading on imbalanced data.
- Precision and recall trade off via threshold.
- Use F1 and ROC to pick the right balance.