Abstract

MRR (mean reciprocal rank) scores each query by $1/\text{rank}$ of its first relevant result and averages those scores. First place scores 1, second 0.5, third 0.33, and a miss 0. Hit@4 asks whether the answer was found. MRR asks how high it ranked.

Cue

• What does MRR measure? → How early the first relevant result appears, averaged over queries
• What does a miss score? → 0
• What does it ignore? → Every relevant result after the first
• Why is it MRR@4 here? → The advisor retrieves only 4 chunks, so ranks beyond 4 are never seen and score as misses

Definition

For a set of queries $Q$, where ${\text{rank}}_q$ is the position of the first relevant result for query $q$:

$$\text{MRR}=\frac{1}{|Q|}\sum _{q\in Q}\frac{1}{{\text{rank}}_q}$$

If no relevant result appears, $1/{\text{rank}}_q=0$. With a cutoff (MRR@k), any first hit beyond rank k also scores 0.

Worked example

The same three questions as the Hit@4 note:

Question	First hit at rank	Reciprocal rank	Hit@4
Q1	1	1.00	1
Q2	3	0.33	1
Q3	none	0.00	0
Mean		0.44 (MRR)	0.67

Q1 and Q2 tie on Hit@4 but not on MRR. Q2’s answer was in the context, but behind two chunks that weren’t.

The reciprocal falls fast: going from rank 1 to rank 2 loses 0.5, but going from rank 3 to rank 4 loses only 0.08. So MRR is mostly a measure of how often the answer comes first.

Rank	1	2	3	4	miss
Score	1.00	0.50	0.33	0.25	0

Implementation (Java)

record Retrieval(List<String> rankedIds, Set<String> relevantIds) {}
 
static double mrrAtK(List<Retrieval> runs, int k) {
    // Unanswerable questions have nothing to find, so leave them out
    var answerable = runs.stream()
            .filter(r -> !r.relevantIds().isEmpty())
            .toList();
    if (answerable.isEmpty()) return 0.0;
 
    double total = 0.0;
    for (var r : answerable) {
        var ranked = r.rankedIds();
        for (int i = 0; i < Math.min(k, ranked.size()); i++) {
            if (r.relevantIds().contains(ranked.get(i))) {
                total += 1.0 / (i + 1);   // ranks are 1-based
                break;                    // only the first hit counts
            }
        }
    }
    return total / answerable.size();
}

Time is $O(|Q|\cdot k)$. It uses the same scan as Hit@k, so compute both in one pass.

Why 4?

In Episode 3 MRR is effectively MRR@4. QuestionAnswerAdvisor uses Spring AI’s default top k of 4, so ranks 5 and below are never retrieved and can only score 0. That cutoff is the right one because the model never sees a chunk ranked 5th. Why 4 is a common choice (screen layouts, LLM context windows, and human attention) is covered in Hit@4 → Why 4?.

A side effect is that the lowest nonzero score is 0.25. Any MRR below 0.25 means most questions missed entirely, not that answers came in low.

Decisions to make before you report it

• What counts as a hit. As with Hit@4, Episode 3 counts only chunks holding the expected element whole. Partial hits don’t count.
• Multiple expected answers. Only the earliest match counts, whichever expected element it is.
• Unanswerable questions. Leave them out of the denominator.
• Ties in similarity score. If two chunks score the same, the store’s ordering decides the rank. Fix the ordering (e.g. by chunk id) so runs are reproducible.

Strengths and blind spots

✅ Sensitive to rank	Separates “found first” from “found fourth”, which Hit@k cannot.
✅ Fits single-answer questions	Most fact questions have one place the answer lives.
❌ Only the first hit counts	A second supporting chunk adds nothing, so it’s a poor fit when the answer needs several chunks.
❌ Steep and nonlinear	Moving from rank 2 to 1 is worth six times as much as moving from 4 to 3. Averages can hide that.
❌ Binary relevance	No partial credit for a chunk that is nearly right. nDCG handles graded relevance.

Related

• Hit@k: did any relevant result appear in the top k?
• Partial@k: Hit@k where holding half the element counts
• Recall@k: what fraction of all relevant results appeared in the top k?
• nDCG@k: a rank-discounted score that credits every relevant result, with graded relevance
• MAP (mean average precision): averages precision at every relevant result’s rank, not just the first
• RAGs to Riches with Spring AI, Episode 3, Knowledge Base

---