Plumloom RAG Evaluation Benchmark
Can a document agent prove it answered from the document? A 20-case corpus-boundary benchmark, matched honest/fabricated controls, second-judge replication, and multi-run Scenario evidence for RAG release decisions.
Across eight matched honest/fabricated trace pairs, every fabricated arm scored below its matched honest arm under DeepSeek V4 Pro (8 of 8). Kimi K3 independently reproduced the same ordering in all eight pairs. Factuality and Relevance carried the strongest separation under both judge configurations.
Resources for this report
Primary question
Can an enterprise team determine whether a RAG agent is operating faithfully within its approved corpus, understand where it fails, and distinguish a repeatable quality signal from one-run evaluation noise?
Enterprise document agents face a deceptively hard evaluation problem. A response can be fluent, accurate in the wider world, and still be wrong for the document the agent was asked to use. The answer may come from retrieved evidence, model memory, or adjacent domain knowledge, and those sources can be difficult to distinguish by reading the final response alone.
We built a 20-case corpus-boundary benchmark using NIST SP 800-160 Vol. 1 Rev. 1, Engineering Trustworthy Secure Systems, as the source corpus. It contains specialized definitions that overlap with familiar cybersecurity concepts while using their own technical meanings, so a model can sound authoritative while crossing the corpus boundary.
A document-grounded RAG agent produced the behavior. Trueline EvalEngine evaluated the resulting trajectories, replicated the main findings with a second judge, and turned one trace-level failure into a focused multi-run Scenario evaluation. The result is a practical model for pre-release RAG evaluation.
Question 1
Is the Agent Inside the Corpus Boundary?
Can evaluation determine whether the answer came from the designated document, and attribute failures to retrieval, evidence use, stopping, or answer generation?
Question 2
Can the Signal Support a Release Decision?
Do trace-level evidence, matched counterfactuals, judge replication, and multi-run statistics separate real failures from evaluation noise?
The benchmark creates difficult corpus-boundary tests. The RAG produces the behavior. EvalEngine turns those executions into trace-level quality evidence, judge-replicated measurements, and multi-run reliability signals.
Nineteen of 20 cases produced the expected answer, refusal, or partial-answer class. One case returned an empty answer even though the required definitions existed in the corpus. Across eight matched counterfactual pairs, every fabricated twin scored below its honest original under both judges: 16 of 16 paired comparisons, with Factuality and Relevance carrying the strongest separation. A multi-run Scenario study of the failed case reproduced its aggregate means within 0.003 and 0.068 across independent replicas.
System under test
Document-grounded RAG agent, BM25 retrieval, retrieve-assess-refine-answer loop
Source corpus
NIST SP 800-160 Vol. 1 Rev. 1 · 195 pages · 473 chunks
Judges
DeepSeek V4 Pro (initial) · Kimi K3 (replication)
Scenario models
MiniMax M3 generation · Kimi K3 judging · temperature 0
Matched controls
8 fabricated twins, retrieval trajectory held fixed
Metrics
Safety · Fluency · Relevance · Factuality · Helpfulness · Completeness
- 20 corpus-verified cases across six behavioral categories
- NIST SP 800-160 Vol. 1 Rev. 1 as the sole source corpus
- 195 PDF pages, 473 deterministic BM25 chunks
- 8 matched honest/fabricated trace pairs
- 2 judge configurations: DeepSeek V4 Pro and Kimi K3
- N16 Scenario study under two controlled information states
- Convergence target CV < 0.1 with 95% confidence intervals
Honest vs. fabricated scores, one row per case
Overall score, faceted by judge. In every row the fabricated twin scores below its honest original: 16 of 16 paired comparisons.
DeepSeek V4 Pro
Kimi K3
The evidence covers one NIST publication, 20 cases, eight matched pairs, two judge families, and one trace-to-Scenario case. It supports claims about this observed experiment. Generalisation to other corpora, agent architectures, or evaluator models would require further replication.
- One corpus, 20 cases, eight matched pairs, and one trace-to-Scenario case. Generalisation to other corpora or agent architectures requires replication.
- The matched fabricated arms use one controlled mutation family: plausible unsupported numerical additions. Other grounding failures may behave differently.
- The second-judge protocol tests robustness to one evaluator change; it establishes judge agreement between DeepSeek V4 Pro and Kimi K3, with broader evaluator families left for future replication.
- The N16 information states differ in both glossary coverage and surrounding-text volume, and the confidence intervals overlap. The comparison supports failure isolation and repeatability; a causal claim would require additional controls.