Abstract

Recall@k is the fraction of a query’s relevant results that appear in its top k, averaged across queries. It is often confused with Hit@k. Hit@k asks whether any relevant result was found, while Recall@k asks how many of them were. The two are equal only when every query has exactly one relevant result.

Cue

• What does Recall@k measure? → The share of each query’s relevant results found in its top k
• How does it differ from Hit@k? → Hit@k scores 1 for any hit, while Recall@k gives partial credit per relevant item
• When are they equal? → When each query has exactly one relevant item
• What’s its upper bound when there are more relevant items than k? → k / (number of relevant items), not 1

Definition

For a set of queries $Q$, where $R_q$ is the set of relevant results for $q$ and ${\text{top}}_k(q)$ is its top k retrieved:

$$\text{Recall@}k=\frac{1}{|Q|}\sum _{q\in Q}\frac{|R_q\cap {\text{top}}_k(q)|}{|R_q|}$$

This is the macro average, where every query counts equally. The micro average pools all items first, ${\sum }_q|R_q\cap {\text{top}}_k(q)|/{\sum }_q|R_q|$, so queries with many relevant items count for more. Say which one you report.

Worked example

Three questions, some with more than one expected element:

Question	Relevant items	Found in top 4	Recall@4	Hit@4
Q1	2	1	0.50	1
Q2	1	1	1.00	1
Q3	3	0	0.00	0
Mean			0.50	0.67

Hit@4 gives Q1 full marks because one of its two elements was found. Recall@4 gives it half. On Q2, which has one relevant item, the two metrics agree.

Implementation (Java)

record Retrieval(List<String> rankedIds, Set<String> relevantIds) {}
 
static double recallAtK(List<Retrieval> runs, int k) {
    // Unanswerable questions have no relevant items, so recall is undefined
    var answerable = runs.stream()
            .filter(r -> !r.relevantIds().isEmpty())
            .toList();
    if (answerable.isEmpty()) return 0.0;
 
    double total = 0.0;
    for (var r : answerable) {
        long found = r.rankedIds().stream()
                .limit(k)
                .distinct()   // a duplicate result shouldn't count twice
                .filter(r.relevantIds()::contains)
                .count();
        total += (double) found / r.relevantIds().size();
    }
    return total / answerable.size();   // macro average
}

Time is $O(|Q|\cdot k)$.

Why 4?

k = 4 because that is how many chunks QuestionAnswerAdvisor puts in the prompt. It is Spring AI’s SearchRequest default. Why 4 is a common choice in general (screen layouts, LLM context windows, and human attention) is covered in Hit@4 → Why 4?.

A small k puts a ceiling on recall. A question whose answer is spread over six chunks can score at most 4/6 = 0.67 at k = 4, even with perfect ranking. Check how many relevant items each question has before reading Recall@4 as a judgement on ranking quality.

Why Episode 3 reports Hit@4, not Recall@4

The post calls Hit@4 “often called recall at k”. The two terms do get used loosely, but in Episode 3 the metric is Hit@4: a question with two expected elements “counts as found if either one is”. That is the right choice there, because:

• Most golden-set questions point at one requirement, so on those questions the two metrics give the same number.
• For a question with two acceptable sources, either one is enough to answer it. Recall would penalise retrieval for not fetching both.

Recall@k is the better metric when a complete answer needs every relevant piece, such as “list every requirement that mentions citations”, or when evaluating a first-stage retriever whose results a re-ranker will narrow down.

Strengths and blind spots

✅ Credits completeness	Rewards fetching every piece of a multi-part answer.
✅ Standard for retriever evaluation	The usual metric for first-stage retrieval before re-ranking.
❌ Ignores rank	Rank 1 and rank k count the same. Pair it with MRR or nDCG.
❌ Ceiling below 1 when there are more relevant items than k	Scores depend on how many relevant items a question has, not just on retrieval quality.
❌ Needs complete labels	Every relevant item must be in the golden set. A missed label lowers recall unfairly.

Related

• Hit@k: did any relevant result appear in the top k?
• Partial@k: Hit@k where holding half the element counts
• MRR: how high did the first relevant result rank?
• Precision@k: the fraction of the top k that is relevant, $|R_q\cap {\text{top}}_k(q)|/k$
• F1@k: the harmonic mean of precision@k and recall@k
• RAGs to Riches with Spring AI, Episode 3, Knowledge Base

---