Ranking & Evaluation Metrics: Precision, Recall, F1, MAP, NDCG

TAKEAWAYS:

• Precision and recall are fundamental metrics that trade off between relevance and coverage in ranked retrieval.
• The precision-recall curve and F1-score balance these metrics for optimal retrieval performance.
• MAP (Mean Average Precision) evaluates ranked lists by averaging precision at each relevant document.
• NDCG (Normalized Discounted Cumulative Gain) accounts for graded relevance in modern search engines.
• Real-world systems like eSewa’s transaction ranking and Daraz’s product recommendations use these metrics to optimize user experience.
• NEPSE’s stock ranking relies on relevance metrics to prioritize financial news and alerts.

---

---

Core Concepts: Precision and Recall

Information retrieval systems return documents in a ranked order. Precision and recall are the two most fundamental metrics used to evaluate how well a system retrieves relevant documents.

Definitions

• Precision (P): The ratio of relevant documents retrieved to the total documents retrieved. $$P=\frac{\text{Number of relevant documents retrieved}}{\text{Total number of documents retrieved}}$$ High precision means fewer irrelevant results.

• Recall (R): The ratio of relevant documents retrieved to the total relevant documents available in the collection. $$R=\frac{\text{Number of relevant documents retrieved}}{\text{Total number of relevant documents in the collection}}$$ High recall means most relevant documents are found, but may include irrelevant ones.

Trade-off Between Precision and Recall

There is an inverse relationship between precision and recall:

• High precision, low recall: Fewer documents retrieved, but most are relevant (e.g., strict filtering).
• Low precision, high recall: Many documents retrieved, but some are irrelevant (e.g., broad search).

Example: Daraz Product Search

Suppose a user searches for "wireless earbuds" on Daraz. The system retrieves 50 products, but only 30 are truly relevant (wireless earbuds). The remaining 20 are related but irrelevant (e.g., wired earphones, headphones).

• Precision (P) = $\frac{30}{50}=0.6$ (60%)
• Recall (R) depends on how many wireless earbuds exist in Daraz’s entire catalog. If there are 100 wireless earbuds in total: R = \frac{30}{100} = 0.3 \text{ (30%)}

Visualization of Precision and Recall:

---

F1-Score: Balancing Precision and Recall

Since precision and recall often conflict, the F1-score combines them into a single metric: $$F1=2\times \frac{P\times R}{P+R}$$

• F1 = 1: Perfect balance (high precision and recall).
• F1 = 0: No relevant documents retrieved.

Example: eSewa Transaction Ranking

eSewa ranks transactions based on relevance. Suppose:

• Precision (P) = 0.7 (70% of retrieved transactions are relevant).
• Recall (R) = 0.6 (60% of all relevant transactions are retrieved).

Then: F1 = 2 \times \frac{0.7 \times 0.6}{0.7 + 0.6} = 0.636 \text{ (63.6%)}

---

Mean Average Precision (MAP)

MAP evaluates ranked retrieval by computing the average precision (AP) for each query and then taking the mean over all queries.

How MAP Works

1. For each query, compute precision at each relevant document (P@k).
2. Calculate the average precision (AP) for the query: $$AP=\frac{1}{R}\sum _{k=1}^n(P@k\times {\text{rel}}_k)$$ where:
   • $R$ = total relevant documents.
   • ${\text{rel}}_k$ = 1 if the $k$-th document is relevant, else 0.
3. MAP is the mean of AP across all queries.

Example: NEPSE Stock News Ranking

Suppose NEPSE ranks stock-related news articles. For a query like "NEPSE share price update", the system retrieves:

1. "NEPSE closes at 1800 points" (Relevant)
2. "Banking sector analysis" (Irrelevant)
3. "NEPSE dividend announcement" (Relevant)
4. "Global market trends" (Irrelevant)

• Precision at 1 (P@1) = $\frac{1}{1}=1.0$
• Precision at 2 (P@2) = $\frac{1}{2}=0.5$
• Precision at 3 (P@3) = $\frac{2}{3}\approx 0.67$

If there are 2 relevant documents, the AP is: $$AP=\frac{1}{2}(1.0+0.67)=0.835$$

If this is the only query, MAP = 0.835.

---

Normalized Discounted Cumulative Gain (NDCG)

NDCG accounts for graded relevance (e.g., some documents are more relevant than others). It discounts irrelevant documents and rewards highly relevant ones early in the ranking.

How NDCG Works

1. Assign relevance scores (e.g., 3 = highly relevant, 2 = relevant, 1 = somewhat relevant, 0 = irrelevant).
2. Compute Discounted Cumulative Gain (DCG): $$DC{G}_p=\sum _{i=1}^p\frac{2^{re{l}_i}-1}{{\log }_2(i+1)}$$
3. Compute Ideal DCG (IDCG) (the best possible DCG for the query).
4. NDCG = $\frac{DCG}{IDCG}$

Example: Pathao Ride Ranking

Suppose Pathao ranks ride options based on user preferences (e.g., price, distance, driver rating). For a query like "cheap ride to Thapathali", the system returns:

1. Driver A: $\text{rel}=3$ (cheap, good rating)
2. Driver B: $\text{rel}=2$ (moderate price, average rating)
3. Driver C: $\text{rel}=1$ (expensive, poor rating)

• DCG at position 3: $$DCG=\frac{2^3-1}{{\log }_2(1+1)}+\frac{2^2-1}{{\log }_2(2+1)}+\frac{2^1-1}{{\log }_2(3+1)}=\frac{7}{1}+\frac{3}{1.585}+\frac{1}{2}\approx 7+1.89+0.5=9.39$$
• IDCG (if the best order were [3, 2, 1]): $$IDCG=\frac{7}{1}+\frac{3}{1.585}+\frac{1}{2}=9.39$$
• NDCG = $\frac{9.39}{9.39}=1.0$ (perfect ranking).

---

Comparison of Evaluation Metrics

---

## In the Real World

1. eSewa Transaction Ranking

   • Idea Used: Precision and Recall
   • How? eSewa ranks transactions to show the most relevant ones first (high precision). However, if a user searches for a specific transaction, eSewa must ensure it retrieves all possible matches (high recall). The system uses F1-score to balance these metrics when optimizing search results.

2. Daraz Product Recommendations

   • Idea Used: Mean Average Precision (MAP)
   • How? Daraz’s recommendation engine ranks products based on user history and preferences. MAP helps evaluate how well the system ranks relevant products early in the list, improving user satisfaction.

3. NEPSE Stock Alerts

   • Idea Used: NDCG
   • How? NEPSE’s alert system assigns graded relevance to news (e.g., a dividend announcement is more relevant than a general market update). NDCG ensures highly relevant alerts appear first, improving investor decision-making.

---

## Exam Tip

1. Understand the Definitions:

   • Memorize the formulas for precision, recall, F1-score, MAP, and NDCG.
   • Know when to use each metric (e.g., MAP for ranked retrieval, NDCG for graded relevance).

2. Practice with Examples:

   • Given a set of documents and queries, compute precision, recall, and F1-score.
   • For MAP and NDCG, practice calculating step-by-step (e.g., precision at each rank, DCG computation).

3. Real-World Applications:

   • Relate metrics to e-commerce (Daraz), fintech (eSewa), and stock markets (NEPSE).
   • Explain how trade-offs between precision and recall affect user experience.

4. Common Pitfalls:

   • Binary vs. Graded Relevance: NDCG is for graded relevance; MAP assumes binary.
   • Order Matters: In ranked retrieval, the position of relevant documents affects MAP and NDCG.

---

Final Note: Always visualize rankings (e.g., precision-recall curves, DCG calculations) in exams to clarify your reasoning. Good luck!