The most useful answer a RAG system can give is sometimes “that is not in this document.” But when asked a question, a model tends to answer anyway, so the honest “not found” has to be built in, not hoped for. Getting the system to say no, and to be right when it does, is harder than getting it to say yes.
This article is a bonus in Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks. It justifies the “I don’t know”: a confident wrong answer is a bug, a bare no-answer is almost as bad, and each of the four bricks has one piece of evidence to show.
🧭 New to the series? Every article in this series sits on our two Towards Data Science author pages, Angela Shi and Kezhan Shi. That is the shortest way to see what is covered and where this one sits.
📓 Runnable companion notebooks are on GitHub: doc-intel/notebooks-vol1.
Ask the corporate chatbot “How much electricity does AI consume globally?” and it returns “I cannot find an answer in this document.” Rephrase and ask again: same answer. At that point most users give up and go to Google.
The system did the right thing. The document it runs on is the World Bank’s Commodity Markets Outlook (April 2025 issue; CC BY 3.0 IGO, as declared on the World Bank Open Knowledge Repository terms of use which states the default license for pre-2023 publications), a 63-page quarterly report on oil, agriculture, and metals prices. AI electricity consumption is not in there. There is no answer to find. The system was not lying.
But the user did not learn anything useful. They cannot tell whether the pipeline looked everywhere it should have, or whether it gave up after one shallow embedding search. “I cannot find an answer” is a verdict that needs to come with a defense. Otherwise it reads as failure, not as a fact.
The series argued that structured output (Article 8) makes “yes” answers verifiable, by forcing the model to cite its evidence: a quote, a page number, a confidence. The same logic applies to “no” answers, and the same four bricks contribute, one piece of evidence each. This article walks the four bricks once more, on a question whose right output is no, and shows what each brick has to produce so the no-answer is defensible.
1. The four bricks, four kinds of evidence
The argument compresses to one table.
The rest of the article walks each row in order, then ties them together on the CMO / AI electricity-consumption case.
2. Parsing: the relational tables are the evidence
The cost of a missing parse is asymmetric. A false positive (extracting noise) makes retrieval messier but is recoverable downstream. A false negative (failing to extract a real token) is silent: the no-answer verdict will look correct, but it is wrong. The answer was there; the pipeline just never saw it.
The good news is that the parsing brick (Article 5B, the relational data model) already produces exactly the data the absence claim needs. The output of parse_pdf is not a wall of text; it is a small relational set of DataFrames: one per kind of thing the document contains. line_df is the central table: one row per text line across the whole document, with its page number, bounding box on the page, and the detected line type (prose, heading, table row, figure caption). The other tables hang off it.
For a no-answer verdict, we do not write a new parsing function. We aggregate over the tables that already exist:
The three places an absence claim still needs to check beyond line_df:
-
Images that carry text: Charts often hold their axis labels and legends as image-embedded text rather than typographic spans. The
image_dfregistry lists every such image; an OCR pass adds anocr_textcolumn to that table without touchingline_df. The parsing brick exposes both before and after states; the coverage report names how many images were OCR’d. -
Tables: A number in a cell parses as a line whose neighbours are the other cells of the same row, not the words above it on the page. If the answer is “AI consumed 460 TWh in 2024” and the figure sits in a table without the word AI in any nearby cell, a sweep on AI misses it.
object_registryflags which pages host tables so the retrieval brick can do a column-aware sweep there (Article 5 section on table parsing). -
Cross-references that did not resolve.
cross_ref_dflists every “see annex B”, “figure 3.D”, “table 11.E” mention in the body. Any row whose target is not inobject_registryis a parsing gap: the body promises an object the extraction did not find. Each gap is a candidate spot where the answer could be hiding.
The parsing brick’s piece of evidence is therefore a small summary derived from the tables it already produces, not a new component. The no-answer verdict can state: “63 pages parsed, 63 with text, 0 unresolved cross-references.” That is enough for the user to trust the parsing layer did its work.
3. Question parsing: enumerate the vocabulary
The retrieval step is only as good as the keywords it sweeps on. If the expert lists AI but forgets artificial intelligence, the retrieval misses every line that spells the concept out. The work of the question-parsing brick is to push the keyword set to exhaustion, with the expert as the source of truth and an algorithmic safety net underneath.
3.1 The expert knows the vocabulary
In enterprise RAG, the expert almost always knows the keywords of their domain. This is the simple case and it covers most questions. They list the concepts that the answer would mention, and for each concept they enumerate the variants: synonyms, acronyms, abbreviations, English / French / German forms if the corpus is multilingual.
For the CMO / AI electricity example, the expert provides this table:
concept_keywords_df schema from Article 6 applied to a ‘no answer’ question – Image by authorThis is exactly the concept_keywords_df schema from Article 6. The same data structure that drives normal retrieval also drives the absence claim. The signature is identical; only the interpretation of the output changes.
3.2 Concept clustering as a safety net
The rare case is the expert who isn’t sure they listed every variant. The safety net is algorithmic: cluster the corpus tokens by embedding similarity, then expose the clusters that the expert’s keywords land in, and ask whether any cluster member should be added.
The output is a list, not a decision. The expert decides what to add. The candidates are a suggestion to consider, not a replacement for the expert’s judgment. The series’ editorial position holds: amplify the expert, do not replace them. The clustering reveals what the expert may not have thought of; the expert validates what belongs in the keyword set.
The keyword set, signed off by the expert, is the question-parsing brick’s piece of evidence.
4. Retrieval: sweep, not top-k
For a normal question (Article 7), retrieval returns top-k: the few pages or lines most likely to contain the answer. For a no-answer claim, that framing fails. The system cannot prove the answer is absent by looking at the top-10 pages. It has to look at every page that mentions any of the concepts.
The shape of the retrieval call therefore changes:
The return value is a DataFrame, not a ranked list. Each row is an EVIDENCE entry: “on page P, line L, the variant V of concept C was found in the snippet S”. The cardinality of the result is what the no-answer verdict turns on:
-
Zero rows for some concept (no variant appears anywhere) → strong evidence the document does not cover that concept at all.
-
Several rows but never co-located (concepts appear on different pages, never together) → medium evidence: the concepts exist but no passage talks about their intersection.
-
Co-located rows (multiple concepts in the same page or paragraph) → weak evidence of absence. Generation will have to look at the snippets and decide whether they answer the question or only touch it.
On the CMO April 2025 case, the result is unambiguous. The full sweep on the keyword set above returns:
The hit-list is the retrieval brick’s piece of evidence. On this question and this document, the verdict writes itself.
5. Generation: structured “no answer” with justification
The last brick takes the parse coverage report, the validated keyword set, and the hit-list, and turns them into the actual response the user sees. The schema mirrors AnswerWithEvidence from Article 8, but the active fields are different.
The generation step does not invent any of the fields. The schema’s first three fields come straight from the previous bricks (parse coverage, the keyword set, the hit-list). The model’s job is narrower: pick the reason, choose which mentions to flag as closest, write the one-sentence “why this is not an answer” for each, and optionally suggest a reformulation.
On the CMO / AI electricity case, the model’s output looks like this:
A user reading this output learns three things at once. The system did look for AI under nine different names and found zero hits anywhere in 63 pages. There is one electricity-related passage, but it concerns India’s coal-driven power demand, not AI. And if they want the answer, the CMO is the wrong corpus to ask. The no-answer is now a useful response.
6. Where this stops
Three edge cases the framework above doesn’t cover cleanly, in order of importance.
Partial answers: A question may have part of its answer in the corpus and part outside it. “How does the EU regulate AI under the AI Act, and how does that compare to U.S. policy?” on an EU-only legal corpus. The retrieval brick will return hits for the EU side and nothing for the U.S. side. The right response is neither a confident yes nor a clean no; it is a structured partial answer that shows what was found and is explicit about what was not. The schema for that is a third sibling of AnswerWithEvidence and AbsenceJustification, with both answer_partial: str and missing_concepts: list[ConceptSearch] fields.
Ambiguous questions: “What about coverage?” asked without context. The system cannot sweep because there is no defined concept set. The right response is a clarification request, not a no-answer. The signal that distinguishes the two is the parsing of the question (Article 6): if the question parser cannot extract concepts, the question is the problem, not the corpus.
Hostile or out-of-scope questions: “What is the meaning of life?” on an insurance-policy corpus. The system can correctly say no-answer, but spending effort on the parse-coverage report and the closest-mentions is wasted. The pipeline should have an upstream “is this question in scope” check that short-circuits the full sweep when the question parser flags the concepts as unrelated to any corpus tag. Pipeline cost matters when this kind of query is frequent.
These cases share a structure: the no-answer schema above is the right shape for absence claims, but it is not the right shape for every “I cannot help” output. Treating it as one of three siblings (yes / partial / no) rather than a single fallback handles the variation cleanly.
7. Conclusion
A defensible “no answer” is not a single sentence; it is a chain of evidence the four bricks produce together. Parsing reports coverage, question parsing reports the expert keyword set, retrieval reports the sweep, generation reports the closest mention and why it does not answer. The user sees the work and can dispute the parse, the keyword set, the closest mention.
The symmetry with the rest of the series is the bigger point. Yes answers are verifiable when the schema forces the model to cite its evidence (Article 8); no answers are verifiable when the schema forces the pipeline to expose its search. It is the same discipline on the same bricks, and both outputs land structured.
8. Sources and further reading
The canonical benchmark establishing “no answer” as a first-class output is Rajpurkar et al. (SQuAD 2.0, ACL 2018). The framing where many questions are unanswerable inside the given passage is Choi et al. (QuAC, EMNLP 2018). The model-side reflection-token analogue of the pipeline-side sweep used here is Asai et al. (Self-RAG, ICLR 2024). The article’s framing: the three-sibling answer schema (yes / partial / no) with a defensible evidence chain on the no-answer branch, parse-coverage report, expert-keyword sweep, closest-mentions, every step auditable.
Earlier in the series:
-
Document Intelligence: series intro. What the series builds, brick by brick, and in what order.
What works, what breaks
-
Baseline Enterprise RAG, from PDF to highlighted answer. The four-brick pipeline end to end: PDF in, highlighted answer out.
-
Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval. Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway.
-
RAG is not machine learning, and the ML toolkit solves the wrong problem. Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead.
-
From regex to vision models: which RAG technique fits which problem. Two axes, document complexity and question control, that pick the technique for each case.
-
10 common RAG mistakes we keep seeing in production. Ten production mistakes, organized brick by brick, with the fix for each.
-
Document parsing
-
Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG. Rebuilding the outline from body typography when the PDF ships no contents page at all: six signals, one bounded loop.
-
Before Full Agentic RAG: Know How You Decide, and the Parsing Methods You Pick From. The parsing methods as a catalogue, and the decision of which to run, before handing the loop to an agent.
Generation
-
Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract. Seven recurring ways a model gets the extraction wrong, and the typed contract that catches each one.
-
Loop engineering for RAG generation: an LLM cascade from a cheap local model up to a hosted flagship. Starting on a cheap local model and escalating only when the answer does not hold up, measured.
One-document pipelines
-
Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model. Cutting a pipeline’s latency and cost by calling the model less often and cheaper, not by buying a faster one.
-
RAG workflow and loop engineering: the dispatcher that decides when to loop and when to stop. Feedback loops, bounded iteration, and the dispatcher, composed into one workflow.
-
Loop engineering for RAG: the small loops inside each step, the big loops across the pipeline. The two scales of loop: small bounded loops inside each brick, big generation-triggered loops across them.
-
One doc to a corpus
-
Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One. Three questions that tell you which shape a document collection has, and the price of building for the wrong one.