ROC-AUC and PR-AUC evaluate binary classifiers across many thresholds, but they emphasize different behavior.
ROC-AUC: Customer Churn
ROC-AUC measures how often the model ranks a random positive above a random negative. Consider a subscription service assigning every customer a churn-risk score. When churned and retained customers are reasonably balanced, ROC-AUC summarizes overall ranking quality across thresholds.
PR-AUC: Rare Positive Events
PR-AUC emphasizes precision and recall for the positive class, making it more informative when positives are rare.
Fraud detection: Suppose a payment platform processes one million transactions containing only 1,000 fraudulent payments. ROC-AUC may look excellent because legitimate transactions dominate. PR-AUC reveals how much fraud the model catches and how many blocked payments are actually fraudulent.
Recommendation systems: Clicked or purchased items are rare among thousands of candidates. PR-AUC shows whether the system retrieves relevant items without overwhelming users with irrelevant recommendations.
Precision@K and Recall@K
Recommendation systems usually display only the top K results, so threshold-independent AUC metrics do not fully describe the user experience.
Precision@K is the fraction of the top K recommendations that are relevant:
Precision@K = relevant items in top K / K
Recall@K is the fraction of all relevant items retrieved in the top K:
Recall@K = relevant items in top K / total relevant items
Suppose a user has 10 relevant movies and 4 appear in the top 5 recommendations. Precision@5 = 4/5 = 0.8, while Recall@5 = 4/10 = 0.4.
Prioritize Precision@K when screen space is limited and every poor result damages trust. Prioritize Recall@K when discovery matters and missing a relevant item is costly. Report both at the actual serving cutoff and average them across users.
NDCG@K
NDCG@K evaluates ranking quality when relevance is graded and position matters. DCG gives more credit to highly relevant items near the top and logarithmically discounts lower positions. NDCG divides DCG by the best possible ordering, producing a score from 0 to 1.
For five recommended movies, placing a five-star movie first scores higher than placing it fifth, even when Precision@5 is identical. Use NDCG@K for recommendation or search systems where users examine top results first and relevance has multiple levels.
MRR
Mean Reciprocal Rank measures how quickly the first relevant result appears. For each query, calculate 1 / rank of the first relevant result, then average across queries. A first relevant result at position four scores 0.25.
NDCG vs MRR: MRR ignores relevant results after the first and usually assumes binary relevance. NDCG evaluates the entire top-K ranking, supports graded relevance, and rewards multiple relevant results. Use MRR when one correct answer satisfies the user, such as question answering. Use NDCG when overall list quality matters, such as recommendations.
Practical Choice
Use ROC-AUC for broad ranking comparisons. Prefer PR-AUC for rare positives, then choose a deployment threshold using business costs and capacity.