Abstract

Hit@k (also written Hits@k or HR@k, for hit rate) is the share of queries where at least one relevant result appears in the top k retrieved. Hit@4 fixes k = 4, the number of chunks the RAG advisor passes to the model by default. It answers one question: did the answer make it into the context window at all?

Cue

• What does Hit@k measure? → The share of queries with ≥ 1 relevant result in the top k
• What does it ignore? → Where in the top k the hit landed, and how many hits there were
• How is it related to MRR? → Hit@k checks whether the answer was found, and MRR checks how high it ranked
• When does Hit@k equal Recall@k? → Only when each query has exactly one relevant item

Definition

For a set of queries $Q$, where $d_{q,r}$ is the result at rank $r$ for query $q$:

$$\text{Hit@}k=\frac{1}{|Q|}\sum _{q\in Q}1\left[\exists r\le k:d_{q,r}\text{ is relevant to }q\right]$$

Each query scores 1 if any of its top k results is relevant and 0 otherwise. Hit@k is the average of those scores.

Worked example

Three questions, with the rank of each one’s first hit:

Question	First hit at rank	Hit@4	Reciprocal rank
Q1	1	1	1.00
Q2	3	1	0.33
Q3	none	0	0.00
Mean		0.67	0.44 (MRR)

Q2 counts as a full hit for Hit@4 even though its answer came in third. MRR is the metric that penalises it for that.

Why 4?

Hit@1, Hit@5 and Hit@10 are also common, but 4 is often chosen for three practical reasons.

1. Screen layouts and “above the fold” design. Vertical space is tight in modern web and mobile UIs.

• Mobile screens: a typical phone shows about 3 to 4 dense search results, product listings or recommendations before the user has to scroll.
• Grid layouts: many desktop interfaces put recommendations in a row of cards, and a standard grid often uses 4 columns (or a 2×2 grid). Hit@4 is a good proxy for whether a relevant item is visible at a glance.

2. LLM context windows. In RAG, the retriever’s k sets the prompt size.

• Feeding in 10 or 20 documents can exceed token limits, increase latency and cost, and cause the “lost in the middle” effect, where LLMs overlook text in the middle of long prompts (Liu et al., 2023).
• The top 4 is a sweet spot. It gives the model enough varied context to answer well while keeping the prompt short, cheap and fast.

3. Human attention and cognitive load. Studies of search behaviour show users rarely look past the first few results. A threshold of 4 measures that “immediate attention zone.” A system with a high Hit@10 but a poor Hit@4 makes users dig too deep to find what they want.

In Episode 3

Episode 3 is the RAG case (reason 2). QuestionAnswerAdvisor uses Spring AI’s SearchRequest default top k of 4. Four chunks of about 800 tokens fit easily in the 16,384-token window set for qwen3:8b. A chunk at rank 5 never reaches the prompt, so Hit@4 measures exactly what the model receives.

The cost of a small k is that every place counts. Close-but-wrong chunks can take one of the four slots, which is why Hit@4 is reported alongside MRR.

Implementation (Java)

record Retrieval(List<String> rankedIds, Set<String> relevantIds) {}
 
static double hitAtK(List<Retrieval> runs, int k) {
    // Unanswerable questions have nothing to find, so leave them out
    var answerable = runs.stream()
            .filter(r -> !r.relevantIds().isEmpty())
            .toList();
    if (answerable.isEmpty()) return 0.0;
 
    long hits = answerable.stream()
            .filter(r -> r.rankedIds().stream()
                    .limit(k)
                    .anyMatch(r.relevantIds()::contains))
            .count();
    return (double) hits / answerable.size();
}

Time is $O(|Q|\cdot k)$ with set lookups. This is the same top-k scan MRR uses, so both metrics usually come out of one pass.

Decisions to make before you report it

• What counts as a hit. In Episode 3 a chunk must hold the expected element whole to count. A chunk holding at least half of it is tracked separately as [[Algos, Information Retrieval, Partial@K|Partial@4]]. The gap between the two shows what cutting chunks by token count costs.
• Multiple expected answers. If a question has two expected elements, it counts as found when either one appears. That is what makes it Hit@k and not Recall@k, which would count the fraction of relevant items found.
• Unanswerable questions. Exclude them from the denominator, since they have no relevant item. Measure abstention on them separately.
• Choice of k. Match k to what the generator actually sees. Hit@10 on a pipeline that only passes 4 chunks overstates retrieval quality. See Why 4?.

Strengths and blind spots

✅ Easy to explain	”In 67% of questions, the answer was in the context.”
✅ Matches RAG’s needs	The LLM reads all k chunks, so getting the answer in at all matters most.
❌ Ignores rank	A hit at rank 4 scores the same as a hit at rank 1. Pair it with MRR or nDCG.
❌ Ignores multiplicity	One relevant chunk scores the same as four.
❌ Saturates	As k grows, Hit@k tends toward 1 and stops telling systems apart.

Related

• Partial@k: the relaxed version, where holding half the element counts
• MRR (mean reciprocal rank): the average of 1 / rank of the first hit, which rewards finding the answer early
• Recall@k: the fraction of all relevant items found in the top k
• Precision@k: the fraction of the top k that is relevant
• nDCG@k: a rank-discounted score that can use graded relevance
• RAGs to Riches with Spring AI, Episode 3, Knowledge Base

---