Evals for RAG

Why eval matters for LLM apps

Regular web app tests are deterministic: input X → assert output Y. RAG is non-deterministic: same question → potentially different answers, both “correct” in different phrasings. Asserting exact strings is useless.

You need metrics that measure quality in aggregate, even when individual responses vary. Without them, every prompt tweak is a guess. With them: “I changed X, faithfulness dropped from 0.95 to 0.78, revert.”

The other reason: LLM calls cost money. Without eval, you can’t compare “this slower but cheaper change kept quality the same” because you have no quality number to compare against.

The golden set: hand-curated test data

The test data for RAG is questions you know the right answer to. Mine lives at eval/golden.jsonl:

{"question": "What is the difference between const and static in Rust?", "expected_answer": "...", "source_file": "rust 2 - constants and static variables.md"}
{"question": "What are the three rules of ownership in Rust?", "expected_answer": "...", "source_file": "rust 8 - ownership OBRM.md"}

Each row:

• question - what to ask
• expected_answer - for human review (not used by faithfulness or answer relevancy; useful for sanity-checking RAG output)
• source_file - which note should be cited

Five entries for v1. Enough to detect regressions across different topics, few enough that eval runs in under 2 minutes and costs ~$0.01.

In production, grow to 30-100 questions across more topic types.

Faithfulness: did the answer stick to the context?

The most important RAG metric. It catches hallucinations.

how it works (Ragas pseudocode):

def score_faithfulness(question, answer, contexts):
    # 1. Decompose the answer into atomic factual claims
    claims = judge_llm.call(f"Break this answer into individual claims: {answer}")
    # claims: ["Each value has one owner", "Owner can only be one at a time", ...]

    # 2. For each claim, ask if the context supports it
    verdicts = []
    for claim in claims:
        verdict = judge_llm.call(f"Context: {contexts}\nClaim: {claim}\nSupported? yes/no")
        verdicts.append(verdict == "yes")

    # 3. Score = fraction supported
    return sum(verdicts) / len(verdicts)

So faithfulness is two LLM judge calls minimum per question (decompose + verify). For 5 questions with ~4 claims each, that’s ~25 judge calls.

Range: 0.0 to 1.0. My RAG app scores 0.95-1.00 because the system prompt says “answer only from the provided context” and gpt-4o-mini obeys.

Why faithfulness scores wiggle

The judge is an LLM. LLMs are non-deterministic. Same answer, same context, same prompt template → slightly different verdicts each run. The decomposition might split a claim one way today and a different way tomorrow.

Net effect: faithfulness wiggles by ~5% between runs. That’s the nature of LLM-judged metrics, not a bug. Report a range (0.95 - 1.00) instead of a single number.

If you need stability:

• Average over 3+ runs (smooths variance)
• Use temperature=0 on the judge (still not fully deterministic)
• Use a more capable judge (gpt-4 instead of gpt-4o-mini) - cleaner verdicts, higher cost

Answer relevancy: did the answer match the question?

Catches off-topic responses. Doesn’t use an LLM as a judge - uses embeddings.

how it works (Ragas pseudocode):

def score_answer_relevancy(question, answer):
    # 1. Generate N questions the answer could be a response to
    hypotheticals = llm.call(f"Generate {N} questions this answer responds to: {answer}")

    # 2. Embed the original question and the hypotheticals
    q_vec = embed(question)
    hypothetical_vecs = [embed(q) for q in hypotheticals]

    # 3. Average cosine similarity between original and hypotheticals
    return mean(cosine(q_vec, hv) for hv in hypothetical_vecs)

If the answer answered the question, the hypothetical questions it could respond to should be similar to the original question. Cosine similarity captures that.

The only non-deterministic step is the hypothetical-question generation. Embeddings + cosine are deterministic. So answer relevancy wiggles by ~1-2% between runs (much less than faithfulness).

The taxonomy: LLM-judged vs embedding-based vs string-match

General split of eval metrics:

For RAG, use a mix:

• LLM-judged for things only an LLM can evaluate (“does this answer follow from this context?”)
• Embedding-based for structural checks (“is this on-topic?”)
• Human review for prod gates before promoting prompts

Other Ragas metrics worth knowing

Beyond faithfulness + answer relevancy, common Ragas metrics:

• Context precision - of the retrieved chunks, how many are relevant? Catches retriever noise. Needs reference in golden set.
• Context recall - of the relevant chunks in the corpus, how many did the retriever find? Catches retriever misses. Needs reference.
• Answer correctness - compared to ground-truth answer, is your answer correct? LLM-judged. Needs reference.
• Context entity recall - do key entities from the context appear in the answer? Niche.

My app’s EvaluationDataset.from_list(...) includes the reference field (the expected_answer from the golden set), so adding those metrics is just metrics=[Faithfulness(), AnswerRelevancy(), ContextPrecision(), ContextRecall()].

What 5 golden questions do and don’t catch

do catch:

• Catastrophic regressions (faithfulness drops from 0.95 to 0.4)
• A/B comparisons between prompt versions
• “Pipeline is broken” smoke test

don’t catch:

• Edge cases the 5 don’t cover
• Statistical “average quality” with confidence intervals
• Coverage across all topic areas of the corpus
• Systematic failure modes (e.g., “always bad at multi-part questions”)

For systematic detection, grow to 30-100 questions across question types. Alternative: synthetic golden sets generated by an LLM from your corpus. Lower quality than hand-curated but enough volume to detect patterns. v2 idea for my app.

The feedback loop pattern

How eval should be used over time:

1. baseline: run eval, record scores. → faithfulness 0.95, relevancy 0.90
2. change something: tweak prompt, try k=6 instead of k=4, change the splitter.
3. run eval again. → faithfulness 0.92, relevancy 0.91
4. decide:
   • Faithfulness dropped 3 points, relevancy went up 1. Net negative? Revert.
   • Or you care more about relevancy. Keep it.
5. Document the experiment.

This converts prompt engineering from “vibes” to “experiments with measured outcomes.” Every change is a hypothesis. Every regression is loud.

For prod CI: run eval on each PR, post the scores, diff against main. My app doesn’t do this yet (single-dev project) but eval/results.json is the artifact that would feed it.

Where the OpenAI calls come from during an eval run

If you watch your OpenAI usage during uv run python -m rag_app.eval, you’ll see:

embedding calls (~20 per 5-question run):

• 5 for retrieval (one per question)
• ~15 for answer relevancy (hypothetical questions × 5 questions)

chat completion calls (~35 per 5-question run):

• 5 for the actual RAG answers
• ~25 for faithfulness decomposition + verification
• 5 for answer relevancy hypothetical generation

Total: ~55 OpenAI calls per eval run, ~$0.01.

The Redis embedding cache catches the retrieval embeddings if the same question is asked twice (rare). It does not catch Ragas’s internal embeddings (those bypass the wrapper). So Redis doesn’t reduce eval cost; it only helps ingest.

What I noticed building my RAG app

• The judge calls dwarf the actual RAG calls in cost. Eval costs ~$0.01 mostly from Ragas’s faithfulness verifications.
• Faithfulness 1.0 on one run and 0.85 on the next, for the same answers, is normal. Don’t chase the wiggle.
• Answer relevancy is the more boring metric - it’s stable, easy to interpret, and rarely surprises you. Faithfulness is the one to watch.
• Ragas 0.4.3 is mid-migration. I had to add a sys.modules shim for a removed langchain-community module. The deprecated from ragas.metrics import Faithfulness path works; the new from ragas.metrics.collections import Faithfulness path is partly broken in 0.4.3. Picked the deprecated path because it actually round-trips through evaluate() cleanly.
• 5 questions felt like nothing while writing the golden set. But “5 questions × 11 LLM calls each = 55 calls” is enough signal to detect any real regression, and it runs in 90 seconds. Smaller than I expected to be useful.