NDCG and MRR explained, with worked examples

Why ranking needs its own metrics

Precision@k and recall@k count how many relevant results are in the top k, but they ignore the order: a relevant answer at rank 1 and one at rank 5 count the same. Users do not read like that. They look at the first few results and stop. Ranking metrics reward systems that put the best results first. MRR does this for the first relevant result, NDCG for all results with their grade of relevance. For the classification side, see confusion matrix, precision and recall.

MRR: mean reciprocal rank

For each query, find the rank of the first relevant result. The reciprocal rank is 1 ÷ that rank, or 0 when no relevant result is returned. MRR = average of the reciprocal ranks over all queries.

In the example, the first relevant result is at rank 1, 3, 2, not found and 1. The reciprocal ranks are 1, 0.333, 0.5, 0 and 1, and the MRR is 2.833 ÷ 5 = 0.567. In Excel, =IFERROR(1/B16,0) turns a rank into its reciprocal and a "none" into 0. With a cut-off, MRR@k only gives credit when the first hit is within the top k. With k = 2, the query whose first hit is at rank 3 also scores 0.

MRR fits tasks with one right answer, such as finding the help article that solves a problem. It ignores everything below the first hit, so it says nothing about whether the other results are useful.

DCG: discounted cumulative gain

NDCG starts from graded relevance, here 3 = perfect, 2 = good, 1 = partly relevant and 0 = not relevant. Each result earns its grade as gain, discounted by its position: discounted gain = relevance ÷ log₂(rank + 1). Rank 1 counts fully (log₂ 2 = 1), rank 3 counts half (log₂ 4 = 2). In Excel: =B2/LOG(A2+1,2).

DCG@k is the sum of the discounted gains of the top k results. In the example the top 5 have grades 2, 0, 3, 1, 0, so DCG@5 = 2 + 0 + 1.5 + 0.431 + 0 = 3.931. On its own the number has no fixed scale: a query with many relevant documents can reach a much higher DCG than one with few.

IDCG and NDCG@k

To get a score between 0 and 1, divide by the DCG of the best possible order, the ideal DCG. Sort all judged documents for the query by relevance, from high to low, with =LARGE($B$2:$B$8,A2), and compute their DCG in the same way. Here the ideal top 5 is 3, 3, 2, 2, 1, so IDCG@5 = 7.141, and NDCG@5 = DCG@5 ÷ IDCG@5 = 3.931 ÷ 7.141 = 0.550.

The ideal list must include every judged relevant document, also those the search did not return. Row 8 holds one such document with grade 3. Built from the six retrieved results only, the ideal top 5 would be 3, 2, 2, 1, 0 and NDCG@5 would rise to 0.690, hiding the fact that the best document was missed.

Worked example: NDCG@5 for one query and MRR for five

Rows 2–7 are the six results returned for one query, with relevance grades as yellow inputs. Row 8 is a relevant document the search missed: it counts for the ideal ranking only. Rows 16–20 hold the rank of the first relevant result for five queries.

Worked example: NDCG@5 for one query and MRR for five

	A	B	C	D	E
1	Rank	Relevance (0–3)	Discounted gain	Ideal relevance	Ideal DG
2	1	2	2.000	3	3.000
3	2	0	0.000	3	1.893
4	3	3	1.500	2	1.000
5	4	1	0.431	2	0.861
6	5	0	0.000	1	0.387
7	6	2	0.712	0	0.000
8	missed	3
9
10	Cut-off k	5
11	DCG@k	3.931
12	IDCG@k	7.141
13	NDCG@k	0.550
14
15	Query	First relevant rank	Reciprocal rank
16	reset password	1	1.000
17	export invoices	3	0.333
18	change plan	2	0.500
19	webhook retries	none	0.000
20	add a user	1	1.000
21	MRR		0.567

Invented queries and relevance judgments for teaching. Real evaluations use dozens or hundreds of judged queries.

Linear or exponential gain

The example uses the relevance grade itself as gain, as scikit-learn's ndcg_score does. Another common form uses 2^relevance − 1, which rewards the top grade much more strongly. Both are valid, but they give different numbers, so state which one you use and never compare NDCG values computed with different gains or different k.

Which metric to use

Use MRR when one good answer is enough and its position matters, as in question answering or navigational search. Use NDCG@k when you have graded judgments and several results matter, as in product search or recommendations. For RAG retrieval, where the language model reads several chunks, look at recall@k next to NDCG, because a chunk that is missing cannot be used. Always average a metric over a fixed set of judged queries, and compare systems on the same set.

Common mistakes

• Building the ideal ranking from the retrieved results only. Relevant documents the system missed belong in the ideal list, or NDCG looks better than it is.
• Dropping queries without a relevant result from MRR. They count as 0. Leaving them out inflates the score.
• Using log₂(rank) as the discount. That is 0 at rank 1 and divides by zero. The discount is log₂(rank + 1).
• Comparing NDCG values with different k or gains. NDCG@5 and NDCG@10, or linear and exponential gain, are different metrics.
• Averaging DCG across queries. DCG depends on how many relevant documents a query has. Average NDCG instead.
• Relying on MRR alone for RAG. MRR only sees the first hit, but an answer may need several retrieved chunks.