Evaluating RAG systems with real evals: quality before demo effect

A RAG demo can look convincing quickly: upload a document, ask a question and receive a fluent answer. A production specialist workflow needs more. The system must retrieve the correct sources, use relevant passages, expose uncertainty and stop safely when evidence is missing. That is what evaluations establish.

Define the task before the metric

Retrieval-augmented generation can describe very different products. Internal search, legal research and customer support have different failure costs. Before selecting metrics, define the task:

• Which sources are allowed and current?
• Which answer format is expected?
• Does every statement require a citation?
• Which failures are acceptable?
• When must a person decide?
• When must the system refuse to answer?

These rules become test cases. A data set should contain frequent work, rare edge cases, contradictory sources, missing information and disallowed requests.

Build test data from real work

Synthetic questions help at the beginning but often mirror the structure that the generating model prefers. Representative cases supplied by specialists are more valuable. They expose shorthand, ambiguity and implicit knowledge from the real workflow.

Version the data set. Otherwise a team cannot explain why a new chunking strategy or model is supposedly better.

Measure retrieval and answers separately

If the correct passage is absent from context, even a strong model cannot answer reliably. Retrieval metrics establish whether relevant documents and passages appear among the first results. Precision, recall, mean reciprocal rank and nDCG can help, depending on the search task.

Answer evaluation then covers:

1. factual and specialist correctness,
2. completeness of required claims,
3. grounding in supplied sources,
4. accurate citations and locations,
5. correct refusal where evidence is missing,
6. compliance with format and policy.

One overall score hides trade-offs. More results may improve recall while increasing irrelevant context, cost and failure risk.

Calibrate model-based evaluation

An LLM can review other model responses efficiently, but it must not become an unexamined source of truth. A small gold set reviewed by specialists shows whether automated judgement agrees with people. Legal, medical and security-sensitive claims need subject-matter oversight.

Blind reviews reduce branding and style bias. Reviewers need explicit criteria and examples. Disagreement is useful because it often reveals an ambiguous product requirement.

Compare changes against a fixed baseline

Chunk size, embeddings, filters, reranker, prompt and generation model all affect output. Changing everything together makes improvement impossible to explain. A clean experiment changes one relevant component and compares:

• quality by case group,
• p50 and p95 latency,
• token and infrastructure cost,
• failure and refusal rates,
• effects on critical cases.

The highest average score is not always the right product decision. A local model with a slightly lower aggregate result may be correct when confidential data rules out external processing and critical cases meet the required threshold.

Continue evaluation in production

A RAG system changes as documents, permissions and user behaviour change. New versions may contradict old statements. Access control, freshness and source deletion therefore belong in ongoing tests.

Production failures become regression cases after sensitive data is removed. Quality and cost are compared for each release. A material decline can stop rollout or restore the previous configuration.

A minimal evaluation record

Thirty to fifty carefully curated cases are more useful than hundreds of automatically generated questions without specialist review. A machine-readable case can look like this:

{
  "id": "contract-termination-014",
  "question": "Which notice period applies to this contract?",
  "permitted_sources": ["contract-2025-04", "terms-v7"],
  "required_claims": ["period", "start date"],
  "forbidden_claims": ["unsupported exception"],
  "must_cite": true,
  "abstain_when_sources_conflict": true,
  "risk": "high"
}

Each release stores retrieval hits, grounding, required claims, forbidden claims, refusal, latency and cost separately. Critical cases receive a hard gate: a better average must not hide a new error in a core legal or security question.

The local versus frontier model decision determines which route runs these evaluations. Once the system calls tools, agent permissions and approvals become part of the same quality boundary.

Sources and further reading

• NIST AI Risk Management Framework
• OWASP Top 10 for LLM Applications
• Information Retrieval Evaluation
• Retrieval-Augmented Generation Paper

Conclusion

RAG quality is not proven by one impressive answer. A versioned data set, separate retrieval and answer metrics, specialist calibration and regression tests make progress measurable. Only then can a demo justify entering a production workflow.