Universal Document Intelligence Platform

What it does

You give it a long document and ask a question. It answers from the document and tells you which page it read, and if the document does not contain the answer it declines instead of inventing one.

The corpus it is measured on is four documents chosen to break it in different ways: Apple’s FY2023 10-K, the original RAG paper, the Bitcoin whitepaper, and GitLab’s FY2026 10-K pulled live from SEC EDGAR as inline-XBRL HTML. Financial tables, academic prose, a protocol spec, and a 2.7 MB filing with zero heading tags in it.

Why the evaluation is the project

A retrieval pipeline is not hard to build any more, and I do not think building one demonstrates much. What is hard is knowing whether yours is any good, and being able to prove it to someone who is not inclined to believe you.

So the thing I actually built is the measurement apparatus, and the pipeline is what it measures. Every number below has a bootstrap confidence interval, a recorded git SHA, and hashes of the exact dataset and corpus that produced it.

A RAG system has two places to fail and they are independent. Retrieval can hand the generator the wrong passages, or the generator can be handed the right ones and still answer badly. A single accuracy number cannot tell those apart, so the harness taps the pipeline twice and scores each stage on its own.

What it scores

Hybrid retrieval, 35 questions across the four documents, gpt-4o-mini for generation and text-embedding-3-large for embeddings:

answer accuracy            94.3%   95% CI [88.6, 98.6]
no-retrieval baseline      40.0%
lift                      +54.3 points
retrieval hit rate         87.1%
faithfulness                0.971
hallucination on unanswerable probes    0.0%
citations pointing to a page shown       100%
latency               1.44s avg, 2.09s p95
cost                  $0.00037 per question

The baseline is the same model answering the same questions with no document at all. It is the number that says whether retrieval did anything, and on the Apple filing it is the difference between 100% and a model guessing at figures it cannot know.

The bug was in my own metric

For a while this project reported that citation accuracy on the live SEC filing was about 29%, against roughly 75% on the PDFs. I wrote it up as an honest weakness of HTML ingestion and moved on.

It was not a weakness. It was a broken measurement, and I only found it because I went back to read all six failures individually instead of trusting the aggregate.

In every one of the six, the retrieved page was right, the answer was right, and the model had cited a different page that also contained the fact. Apple’s revenue appears in the Business overview on page 10 and again in the MD&A discussion on page 93. My answer key listed only the MD&A page. The metric was measuring agreement with my gold set, not whether the citation was truthful.

The tempting fix is to add the other pages to the answer key. That is metric gaming: you widen the key until the number goes up, and you have learned nothing about the system. The real fix is a metric that does not need a curated key at all, so I added citation-supported percentage, which asks whether every page the model cited was one of the pages it was actually shown. That catches fabrication, which is the failure anyone genuinely cares about, and it needs no human labelling. It measures 100% across all 35 questions.

The old gold-based number stays in the report, relabelled as a strict lower bound at 61.3%, because deleting an inconvenient measurement is its own kind of dishonesty. The CI gate now fails below 95% supported.

The tool shows the same thing live. Asked for GitLab’s fiscal 2026 revenue it answers from page 10, while my key lists page 94, and both pages carry the figure:

Those section labels are there because the filing has no pages. SEC primary documents are one continuous HTML stream, so ingestion splits them into synthetic pages and anchors each one to the nearest Item heading, which is the unit a person actually cites a filing by.

Choosing retrieval by measurement

Four strategies, same corpus, same questions:

strategy	retrieval hit	answer accuracy	citation accuracy	latency
dense	77.4%	87.1%	64.5%	1.23s
BM25	64.5%	87.1%	45.2%	0.92s
hybrid	87.1%	94.3%	61.3%	1.24s
hybrid + rerank	93.5%	94.3%	83.9%	2.34s

Hybrid fuses dense and BM25 by Reciprocal Rank Fusion, and it wins because the two fail differently. Dense embeddings blur short specific strings, so BM25 catches the needle facts, while dense catches the paraphrases BM25 cannot match.

Reranking is the interesting row. It does not improve answer accuracy at all, it ties. What it buys is retrieval hit rate and a large jump in citation accuracy, for double the latency. So it is not the default, it is a documented choice between a fast answer and a well-sourced one.

And the honest caveat, which is in the README and the report as well as here: at 35 questions, hybrid’s 7.2 point gain over dense is not statistically significant, p is about 0.19, and the confidence intervals overlap. The retrieval-level gain is real and the downstream answer gain is not yet proven. The question set has to grow before I will claim otherwise, and I would rather publish the caveat than the headline on its own.

Chunking, where there was no winner

I swept chunk size expecting a best setting to fall out. It did not:

chunking	answer accuracy	citation precision	recall@k
256 / 32	94.4%	66.7%	0.875
512 / 64	92.6%	79.2%	0.833
1024 / 128	96.3%	66.7%	0.833

Retrieval hit rate was flat at 91.7% across all three. Bigger chunks answer better because the model gets more context, smaller chunks locate better because there is less irrelevant text attached to each hit, and the two are in genuine tension. There is no dominant setting, so the repository ships the evidence and says the choice depends on whether you are optimising for answers or for provenance.

Being able to see inside a query

The pipeline is also a small FastAPI service, and every query comes back with more than an answer: the pages it cited, the passages it was given, what it cost, and a span trace of the run. The interface is built on my own element collection, so it inherits the same tokens as this site rather than carrying a second look.

The trace splits the same way the evaluation does, into retrieval and generation, which means a slow or wrong answer can be attributed to a stage rather than guessed at. Here retrieval is most of the two and a half seconds, because hybrid search runs a dense query and a keyword query and fuses them.

Making the numbers trustworthy

Measurement is only worth something if it is hard to fool, including by me.

• Determinism. Temperature 0 on generation, baseline, judge, and reranker. Verified by running the same question three times and getting one answer.
• Provenance. Every run records the git SHA plus content hashes of the dataset and corpus, so a number can always be traced to the code and data that produced it.
• Confidence intervals and significance tests, bootstrapped, on every headline metric, with the paired test run on the difference between configurations rather than read off overlapping intervals.
• A CI regression gate with thresholds on accuracy, faithfulness, hallucination, citation support, latency, and cost. I verified it genuinely fails, exit code 1, by breaching a threshold on purpose. A gate that cannot fail is decoration.
• Prompt-injection probes. Three attempts to make the system follow instructions embedded in a retrieved document, three resisted.
• A grounding probe set. Questions the corpus provably cannot answer. The system declines all of them. The no-retrieval baseline confidently invented the 21 million Bitcoin cap.

61 tests, Ruff clean, and I verified the whole pipeline by cloning the repository fresh and running it from scratch, which is how I caught that .gitignore was excluding the results file the CI gate reads.

Swapping the model out from under it

The generator is a setting now, not a hard-wired client, so I can point the whole harness at a different model and change nothing else. I ran IBM’s Granite 3.3 8B locally against gpt-4o-mini: same corpus, same retrieval, same prompt, same temperature, and the same judge grading both sides.

                                  granite3.3:8b     gpt-4o-mini
answer accuracy                   92.9%             94.3%
no-retrieval baseline             30.0%             38.6%
retrieval lift                    +62.9 pts         +55.7 pts
citations to a page actually shown 95.0%            100%

The accuracy gap is not a result. 1.4 points sits inside this harness’s own run-to-run movement: the identical gpt-4o-mini configuration has returned 94.3% and 95.7% on different days, with baselines of 38.6%, 40.0% and 42.9% on byte-identical data. So the honest sentence is that an 8B model running on my laptop answered this corpus about as well as the hosted one, not that it lost.

The difference that is real is citation. Granite pointed at a page it had never been shown 5% of the time. gpt-4o-mini never did. That is the metric I built precisely because it needs no answer key and catches fabrication rather than disagreement, and it is the number that would actually decide which model you deploy.

Retrieval mattered more to the smaller model, which is the part I did not expect. Granite knew 8.6 points less from memory, so the same retrieval bought it more: +62.9 against +55.7. A model with less in its head gains more from being handed the document.

Finding this also taught me something about my own instrument. I had verified determinism by asking one question three times and getting one answer. Across all 35 questions the aggregate still drifts a few points between runs, and that drift is now the bar any future comparison has to clear before it means anything.

What is still open

The question set is too small. 35 questions is a seed set, not a benchmark, and it is the reason the hybrid-over-dense result cannot be claimed. Growing it is the single highest-value thing left, and I write the questions myself rather than generating them, because a model-written eval set tends to ask exactly what a model finds easy.

The judge is not yet validated. Answers are graded by an LLM, and I have the harness to measure its agreement with human labels as Cohen’s kappa plus a confusion matrix, but I have not sat down and labelled the subset. Until I do, every accuracy figure here rests on an ungraded grader.

The chunking decision is unmade, deliberately. The evidence is in the repository and the tradeoff is real, and I would rather leave it open than pick one and pretend the sweep pointed at it.