Algos, Information Retrieval, Partial@K

Abstract

Partial@k is the share of queries where at least one of the top k results holds at least half of an expected answer. It is a relaxed Hit@k, which needs the whole answer in one result. A full hit also counts as a partial one, so Partial@k is never below Hit@k. The gap between them is the cost of chunking: answers the retriever found, but in pieces because a chunk boundary cut through them.

Cue

What counts as a partial hit? → A result holding ≥ 50% of the expected element’s text

Does a full hit count as partial? → Yes, so Partial@k ≥ Hit@k always

What does Partial@k − Hit@k measure? → Questions where retrieval found the right place but the chunking split the answer

Why not just report Partial@k? → Half a requirement can drop the clause that matters, such as “MUST NOT … unless”

Definition

For each query $q$ with expected elements $E_{q},$ let $cov (e, d)$ be the fraction of element $e ‘s$ text that result $d$ contains. Then:

Partial@ k = \frac{1}{∣ Q ∣} q \in Q \sum 1 [\exists r \leq k, e \in E_{q} : cov (e, d_{q, r}) \geq \frac{1}{2}]

Hit@k is the same formula with the threshold at $1,$ not $\frac{1}{2} .$ Because of that, every full hit is also a partial hit:

0 \leq Hit@ k \leq Partial@ k \leq 1

Reading the gap

Hit@4	Partial@4	What it means	What to try
High	High	Retrieval works and chunks keep answers whole.	Nothing here. Look at generation.
Low	High	Chunking is the problem. The right passage is found but cut in two.	Larger chunks, overlap between chunks, or splitting by document structure
Low	Low	Retrieval is the problem. The right passage isn’t found at all.	A better embedding model, query rewriting, hybrid search (BM25 + vectors), or a larger k

The Episode 3 archive shows why the gap exists. TokenTextSplitter cuts every 800 tokens without regard to structure. Of 1,881 requirements, about 1,720 land whole in one chunk and about 160 are cut. A question about a cut requirement can’t score a full hit however good the embeddings are. The best it can do is a partial hit.

Worked example

Three questions, each expecting one requirement:

Question	Best chunk in top 4 holds	Hit@4	Partial@4
Q1	100% (rank 1)	1	1
Q2	60% (rank 2), cut by a chunk boundary	0	1
Q3	20% (rank 3)	0	0
Mean		0.33	0.67

Q2 is a chunking loss. Retrieval found the right place, but no chunk holds the whole requirement. Q3 is a retrieval loss, since 20% is below the threshold.

Measuring “half”

Your text needs a definition of coverage. GoldenScorer in the rags repo uses character overlap on normalized text:

Read the page text exactly as the ingest did (jsoup body().text()), so every chunk is a substring of it.
Find the element’s span $[e_{0}, e_{0} + ∣ e ∣)$ and the chunk’s span $[c_{0}, c_{0} + ∣ c ∣)$ in the page text.
The overlap is $min (e_{0} + ∣ e ∣, c_{0} + ∣ c ∣) - max (e_{0}, c_{0}) .$
If overlap ≥ $∣ e ∣$ it’s a full hit. If $2 \cdot overlap \geq ∣ e ∣$ it’s partial. Otherwise it’s a miss.

Two details matter:

Repeated text. If the element’s text appears more than once on the page, for example in a summary table, judge the chunk by the occurrence it overlaps most.
Fallback. If the chunk isn’t a clean substring of the page (a tokenizer split a multi-byte character, say), check whether the chunk contains either half of the element’s text.

Implementation (Rust)

The interesting part is grading one chunk against one expected element. Everything else (first partial rank, the share at k) is the same bookkeeping as Hit@k.

use std::ops::Range;
 
/// How much of an expected element one chunk holds. Ordered, so `max()` picks the best.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub enum Hit {
    None,
    Partial,
    Full,
}
 
/// Grades `covered` bytes out of an element `len` bytes long.
fn grade(covered: usize, len: usize) -> Hit {
    if covered >= len {
        Hit::Full
    } else if covered * 2 >= len {
        Hit::Partial
    } else {
        Hit::None
    }
}
 
/// Length of the overlap of two half-open spans, or 0 if they don't meet.
fn overlap(a: &Range<usize>, b: &Range<usize>) -> usize {
    a.end.min(b.end).saturating_sub(a.start.max(b.start))
}
 
/// Every start of `needle` in `haystack`, overlapping matches included.
fn occurrences<'a>(haystack: &'a str, needle: &'a str) -> impl Iterator<Item = usize> + 'a {
    let step = needle.chars().next().map_or(1, char::len_utf8);
    let mut from = 0;
    std::iter::from_fn(move || {
        let i = from + haystack.get(from..)?.find(needle)?;
        from = i + step;
        Some(i)
    })
}
 
/// How much of `element` the `chunk` holds, measured as positions in `page`,
/// the text both were cut from.
pub fn classify(page: &str, element: &str, chunk: &str) -> Hit {
    if element.is_empty() || chunk.is_empty() {
        return Hit::None;
    }
    let Some(c0) = page.find(chunk) else {
        return fallback(element, chunk);
    };
    let chunk_span = c0..c0 + chunk.len();
    // The element can appear more than once (a summary table, say): judge by the best overlap.
    let best = occurrences(page, element)
        .map(|e0| overlap(&(e0..e0 + element.len()), &chunk_span))
        .max()
        .unwrap_or(0);
    grade(best, element.len())
}
 
/// The chunk isn't a clean slice of the page, so positions can't be compared.
/// Settle for: does it hold the element, or either half of it?
fn fallback(element: &str, chunk: &str) -> Hit {
    if chunk.contains(element) {
        return Hit::Full;
    }
    let half = (element.chars().count() / 2).max(1);
    let mid = element.char_indices().nth(half).map_or(element.len(), |(i, _)| i);
    let (head, tail) = element.split_at(mid);
    if chunk.contains(head) || (!tail.is_empty() && chunk.contains(tail)) {
        Hit::Partial
    } else {
        Hit::None
    }
}
 
#[cfg(test)]
mod tests {
    use super::*;
 
    const PAGE: &str = "Intro. FR-001: The index is never the source of record. Outro.";
    const FR: &str = "FR-001: The index is never the source of record.";
 
    #[test]
    fn grades_by_how_much_of_the_element_survives_the_cut() {
        assert_eq!(classify(PAGE, FR, PAGE), Hit::Full);
        assert_eq!(classify(PAGE, FR, "Intro. FR-001: The index is never the"), Hit::Partial);
        assert_eq!(classify(PAGE, FR, "Intro. FR-001: The"), Hit::None);
    }
 
    #[test]
    fn repeated_element_takes_best_occurrence() {
        let page = format!("{FR} | {FR}");
        let chunk = &page[page.len() - FR.len()..];
        assert_eq!(classify(&page, FR, chunk), Hit::Full);
    }
 
    #[test]
    fn best_hit_across_the_top_k() {
        let top_k = ["Outro.", "Intro. FR-001: The index is never the", PAGE];
        assert_eq!(top_k.iter().map(|c| classify(PAGE, FR, c)).max(), Some(Hit::Full));
    }
}

Notes on the Rust:

Hit derives Ord (None < Partial < Full), so the best grade across the top k is just .max().
saturating_sub makes non-overlapping spans come out as 0, not as an underflow.
occurrences finds overlapping matches. str::match_indices skips them, so it would miss a repeat that starts inside the previous one. It steps by the first character’s UTF-8 length to stay on a character boundary.
Lengths are in bytes, not characters. Element and overlap are measured in the same unit, so the ratio still works. On non-ASCII text the 50% line can shift slightly compared with a character count. The fallback splits on a character boundary, because split_at panics mid-character.

Choice of k and threshold

k follows the same reasoning as Hit@k: match it to what the generator sees. See Hit@4 → Why 4?. Always report Partial@k and Hit@k at the same k, or the gap means nothing.
The 50% threshold is a convention, not a law. Lower it and the metric drifts toward “retrieved the right neighbourhood”. Raise it toward 100% and it becomes Hit@k. Pick one, say what it is, and keep it fixed across runs.


✅ Diagnostic	Separates chunking failures from retrieval failures, which Hit@k alone can’t.
✅ Cheap	Uses the same top-k scan as Hit@k and MRR, so one pass gives all three.
❌ Half isn’t enough to answer	The missing half can hold the condition or exception, so a partial hit can still produce a wrong answer. Don’t report Partial@k as answer quality.
❌ Ignores which half	50% from the start of a requirement and 50% from the end count the same, though one may hold the key clause.
❌ Needs span-level labels	You need the expected element’s exact text and position, not just a document id.

Hit@k: the strict version, which needs the whole element in one result
MRR: how high the first full hit ranked
Recall@k: the fraction of all relevant items found
RAGs to Riches with Spring AI, Episode 3, Knowledge Base

Algos, Information Retrieval, Partial@K

Definition

Reading the gap

Worked example

Measuring “half”

Implementation (Rust)

Choice of k and threshold

Strengths and blind spots

Related