Attest
Grounded generation is not complete when a model cites something. It is complete when the system can verify the claim, measure the failure, and know when to ask for help.
Builder · Retrieval, verification, evaluation, and backend reliability
Incoming claim
The agreement renews annually unless written notice is provided.
Retrieved evidence
2 passagesEvidence supports annual renewal with 60 days' written notice.
4.15%
failing citations after verification
0.473
evidence F1 on Qasper
41
questions in benchmark
Problem
RAG systems can produce fluent answers whose citations do not actually support the claim being made. That makes a high-level answer score a poor proxy for whether a user can safely act on it.
Why it matters
Evidence quality is a systems problem: retrieval, generation, verification, revision, and escalation all need explicit boundaries so uncertainty remains visible instead of becoming confident prose.
Architecture
7 stages · select to inspect
Figures from the evaluation
Technical challenges
Verifying the exact claim
The system evaluates claim-to-passage support rather than treating the presence of a citation as proof of grounding.
Combining retrieval signals
pgvector similarity and PostgreSQL full-text search are fused before reranking so exact terms and semantic matches can both survive retrieval.
Making uncertainty actionable
Explicit metrics distinguish supported answers, unsupported claims, and cases that should be escalated rather than revised indefinitely.
Tradeoffs
Independent judge over self-checking generation
A separate verifier creates a meaningful failure boundary instead of asking the same generation path to grade itself.
Bounded revision over open-ended reflection
A fixed retry budget makes latency and cost predictable while preserving a clear escalation path.
Evidence metrics over answer fluency
The benchmark rewards support and calibrated failure, not only a readable final sentence.
Experiments
- 01Compared citation failure rates before and after claim-level verification on a 41-question benchmark.
- 02Evaluated hybrid retrieval and reranking against semantic retrieval alone.
- 03Measured evidence F1 on human-annotated Qasper examples.
Results
Reduced failing citations from 9.87% to 4.15% after verification.
Reached 0.473 evidence F1 on the human-annotated Qasper benchmark.
Produced explicit escalation signals for unsupported answers and uncertain cases.
Lessons learned
- A citation is a pointer; verification tests whether it actually supports the claim.
- Retrieval quality and evaluator quality are coupled system dependencies.
- Human escalation is a reliability feature when the system knows what it cannot prove.
Future work
- Expand claim decomposition for multi-part answers.
- Publish a reproducible citation-verification benchmark.
- Track verifier calibration and cost-quality tradeoffs across providers.