Picture a folder of eight arXiv papers, about 129 pages in all, and one question: “Which model performs best on the Ubuntu dataset for text lengths between 60 and 90 words?” The answer is a single cell in a single table on a single page of one of those papers. Nothing in the running text says it out loud. To get it right you have to find the right paper, find the right page, find the right column, and then read a number off a table.

Most document QA demos skip all of that by starting from one PDF. Our NAACL 2025 paper, VisDoM, is about the version of the problem that analysts, scientists and lawyers actually have: a question over a collection of documents where the evidence is usually visual. We built a benchmark for it (VisDoMBench) and a retrieval-augmented method that answers it (VisDoMRAG). This post traces that Ubuntu question through VisDoMRAG one stage at a time.

The idea in one picture

A retrieval-augmented generation (RAG) system for PDFs has to pick a modality. Retrieve text and the model reads tables as flattened OCR strings. Retrieve page images and the model sees the table as pixels, which is better for tables and charts but leaves it alone with whatever it misreads. Our answer is to do both, in two independent pipelines, and then make a language model check the two reasoning chains against each other before committing.

Text-only RAG PDFs, OCR'd into text chunks retrieve chunks, LLM reads text table rows arrive as flattened OCR text 37.3 GPT-4o avg. on VisDoMBench Visual-only RAG PDFs, rendered as page images retrieve pages, LLM looks at pixels sees the table, but may misread a cell 49.0 GPT-4o avg. on VisDoMBench VisDoMRAG (ours) PDFs, as page images and as text chunks visual branch textual branch compare the two reasoning chains 50.0 GPT-4o avg. on VisDoMBench Long context (every page in the prompt) scores 32.8 with the same model.
Three ways to answer the same question over a document collection. The averages are GPT-4o's end-to-end accuracy across the five VisDoMBench splits (Table 3 of the paper).
Key ideaRun a visual RAG pipeline and a textual RAG pipeline in parallel, make each one write down its evidence and its reasoning, and let a final LLM call resolve the two chains (consistency-constrained modality fusion). The fusion step is where a number read off a table image gets reconciled with the OCR text that describes it.

Walkthrough: one question, 129 pages

Walkthrough: "Which model performs best on the Ubuntu dataset for text lengths between 60 and 90 words?"

1. The pile of PDFs and the question

Figure 1 of the paper. In single-document QA (top) the grounding context is somewhere in the one PDF you were handed. In multi-document QA (bottom) the same question has to be answered from a collection: the KEHNN text-matching paper plus distractor papers on graph matching, time-series clustering, visual grounding, appearance maps and robust loss functions.

This is a PaperTab-style query from VisDoMBench. The document that answers it is “Knowledge Enhanced Hybrid Neural Network for Text Matching” (the KEHNN paper), and the evidence is its Table (b), the Ubuntu dataset results. Every query in the benchmark comes with distractor documents, so the collection spans roughly 50 to 200 pages and the system has to locate the document before it can locate the table.

2. Index the collection twice, in parallel

Document collection 8.4 documents, ~129 pages Page images ... one image per page, no OCR Visual index ColQwen2 embeds each page image OCR text chunks PyTesseract, 3,000 chars, 10% overlap Text index BGE-1.5 embeds each chunk keeps doc + page metadata

VisDoMRAG never chooses between pixels and text; it builds both indexes. The visual branch renders every page as an image and embeds it with a late-interaction visual retriever (ColQwen2; ColPali also works), with no OCR at all. The textual branch OCRs every page with PyTesseract, splits the text into 3,000-character chunks with 10% overlap (recursive split), keeps the source document and page number as metadata, and embeds each chunk with BGE-1.5. Both indexes are built once per collection.

3. Retrieve in both modalities

Query Which model performs best on the Ubuntu dataset for text lengths between 60 and 90 words? Visual retriever (ColQwen2), top-5 pages KEHNN paper, page with Table (b) Ubuntu dataset KEHNN paper, another page Time Series Clustering paper Graph Matching paper Deep Appearance Maps paper k=5 Text retriever (BGE-1.5), top-7 chunks OCR of Table (b): "KEHNN 0.724 0.774 0.785 ..." KEHNN paper, experiments section Robust Loss Function paper Visual Grounding paper Time Series paper ... two more chunks k=7

The same question is scored against both indexes. The visual retriever returns the top-5 pages, and the page of the KEHNN paper that carries Table (b) is among them; so are pages from distractor papers, which is what makes the later curation step necessary. The textual retriever returns the top-7 chunks, one of which is the OCR of that same table, flattened into a string like “KEHNN 0.724 0.774 0.785 0.791”. We picked k=5 and k=7 as the smallest windows whose retrieved evidence overlapped the ground-truth evidence at ANLCS 0.7 or better (Figure 3 of the paper).

4. Each branch curates evidence, reasons, then answers

1. Evidence curation 2. Chain of thought 3. Answer Visual branch 5 page images Table (b) Ubuntu, column [60, 90): LSTM 0.732 MV-LSTM 0.725 KEHNN 0.785 read off the page image "60 to 90 words" is the [60, 90) column. The largest value in it is 0.785, which belongs to the KEHNN row. KEHNN Textual branch 7 text chunks From the OCR chunk, the row "KEHNN 0.724 0.774 0.785 0.791" under header "[0,30) [30,60) [60,90) [90,inf)" verbalized from flattened text Third column is [60, 90). Third values: LSTM 0.732, MV-LSTM 0.725, KEHNN 0.785. KEHNN is highest. KEHNN

Neither branch is asked for an answer straight away. The LLM is prompted three times: first to pull out and write down the evidence that matters (evidence curation), then to link that evidence into an explicit argument (chain-of-thought reasoning), and only then to answer in the format the question type calls for (answer generation). In the visual branch the multimodal LLM reads the [60, 90) column off the page image: LSTM 0.732, MV-LSTM 0.725, KEHNN 0.785. In the textual branch the LLM has to recover the same column from the flattened OCR row. Curation is what stops the distractor pages and chunks from leaking into the reasoning.

5. Fuse the two chains with a consistency check

Visual branch evidence + chain + answer Textual branch evidence + chain + answer Modality fusion Are the two reasoning chains consistent? one LLM call over both branches' outputs Chains agree: confirm Ubuntu question: KEHNN vs KEHNN, same cell, same column, same row final answer: KEHNN Chains conflict: re-examine StackEx question: visual says 57.5%, textual says 298k (the train split only) Fusion re-reads the table and sums final answer: 330k (train + valid + test)

A final LLM call receives the curated evidence, the reasoning chain and the answer from both branches and is asked whether the two chains are consistent (modality fusion). When they agree, as they do for the Ubuntu question, the answer is confirmed with two independent lines of evidence behind it. When they conflict, the LLM has to re-examine the evidence and reconcile the difference. The paper’s PaperTab example (Figure 5) shows the conflict case: asked for the size of the StackEx keyphrase dataset, visual RAG latched onto 57.5% and textual RAG onto 298k, both wrong numbers from the right table; the fused reasoning summed the splits and answered around 330k.

6. The final answer, with its evidence attached

query→5 pages7 chunks→visual chaintextual chain→fusion→KEHNN

The curated evidence survives to the output, so the answer arrives with the table it came from. Rebuilt from Table (b) of the KEHNN paper (the grounding context in Figure 1):

Length [0, 30) [30, 60) [60, 90) [90, ∞)
#Pair 253578 207772 33618 5032
LSTM 0.707 0.748 0.732 0.718
MV-LSTM 0.726 0.752 0.725 0.694
KEHNN 0.724 0.774 0.785 0.791

Final answer: KEHNN. Because both branches wrote their evidence down before answering, the response is verifiable: you can point at the cell that produced it (the paper calls this implicit context attribution).

Under the hood

Symbol Meaning
\(q\) the query
\(\mathcal{D} = \{d_1, \dots, d_n\}\) the document collection for that query (8.4 documents, 129 pages on average)
\(\mathcal{P}\), \(\mathcal{C}\) all page images, and all OCR text chunks, of \(\mathcal{D}\)
\(\mathbf{q}_i\), \(\mathbf{p}_j\) token-level embeddings of the query and of a page image (late-interaction retriever)
\(k_v = 5\), \(k_t = 7\) number of retrieved pages and chunks
\(E_m, R_m, a_m\) curated evidence, reasoning chain and answer of branch \(m \in \{v, t\}\)
\(f_\theta\) the (frozen) LLM, prompted differently at each step

Retrieval. The visual retriever scores a page by late interaction (MaxSim): every query token embedding is matched to its most similar patch embedding of the page, and the similarities are summed.

\[s(q, p) = \sum_{i=1}^{|q|} \max_{j} \, \langle \mathbf{q}_i, \mathbf{p}_j \rangle, \qquad \mathcal{P}_q = \operatorname{top\text{-}k_v}_{p \in \mathcal{P}} s(q, p)\]

The textual retriever is a dense bi-encoder (BGE-1.5), scoring chunks by embedding similarity to give \(\mathcal{C}_q\), the top-\(k_t\) chunks. We benchmarked BM25, MiniLM, MPNet and BGE-1.5 on the text side and ColPali and ColQwen2 on the visual side; ColQwen2 and BGE-1.5 were the best of each and are what the end-to-end numbers use.

Per-branch prompting. Each branch is the same three-call chain, applied to its own context:

\[E_m = f_\theta(q, \mathcal{X}_m;\ \text{curate}), \quad R_m = f_\theta(q, E_m;\ \text{reason}), \quad a_m = f_\theta(q, E_m, R_m;\ \text{answer}), \qquad \mathcal{X}_v = \mathcal{P}_q,\ \mathcal{X}_t = \mathcal{C}_q\]

The curation call is the one that does the multi-document work: it must isolate the paragraphs, table rows or figure details that bear on \(q\) and verbalize them in a structured form, dropping the distractor content that retrieval let through.

Consistency-constrained fusion. The final answer is one more LLM call over everything both branches produced, with the instruction to judge whether \(R_v\) and \(R_t\) are consistent and to reconcile them if not:

\[a = f_\theta\big(q, (E_v, R_v, a_v), (E_t, R_t, a_t);\ \text{consistency}\big)\]

This is a late-fusion design, in the spirit of self-consistency over chains of thought: each modality is processed independently and the chains are compared afterwards. The early-fusion alternative, appending the OCR text of the visually retrieved pages to the image prompt, scores 43.63 average with GPT-4o against 50.01 for late fusion.

q, D query and collection of n documents Visual RAG branch page images ColQwen2 late interaction top k=5 pages multimodal LLM, three prompts curate evidence E_v reason step by step R_v answer a_v Textual RAG branch OCR + chunks BGE-1.5 3,000-char chunks top k=7 chunks LLM, same three prompts curate evidence E_t reason step by step R_t answer a_t Fusion consistency check on R_v vs R_t late fusion a
VisDoMRAG end to end. Grey boxes are retrieval and prompting steps shared with unimodal RAG; the fusion step is the contribution. Retrieval is bounded (5 pages and 7 chunks), so the context handed to the LLM does not grow with the size of the collection.

The benchmark. VisDoMBench re-purposes five datasets that have public source documents and grounded evidence, de-duplicates questions across splits, drops trivial ones, and augments each question with distractor documents. Ambiguous questions (common in PaperTab and SciGraphQA) are rewritten by GPT-4o into more specific variants and a human annotator picks one, keeps the original, or discards the item, so that exactly one document answers each question.

Split Content Queries Docs Avg. docs / query Avg. pages / query
PaperTab tables, text (scientific papers) 377 297 10.82 113.10
FetaTab tables (Wikipedia) 350 300 7.77 124.33
SciGraphQA charts (scientific papers) 407 319 5.91 129.71
SPIQA tables, charts (scientific papers) 586 117 9.51 135.58
SlideVQA slides (presentation decks) 551 244 6.99 139.71
VisDoMBench tables, charts, slides, text 2,271 1,277 8.36 128.69

Answers are scored with word-overlap F1 (the UDA variant for PaperTab, which handles binary and short-text answers). A retriever is credited with identifying the source document when at least \(\lceil k/2 \rceil\) of its top-\(k\) results come from the ground-truth document.

What the numbers say

Method LLM PaperTab FetaTab SciGraphQA SPIQA SlideVQA Average
Long context GPT-4o 28.37 60.03 24.12 36.30 15.06 32.78
Text RAG GPT-4o 37.34 60.82 29.74 42.80 15.97 37.33
Visual RAG GPT-4o 42.01 61.89 31.12 43.28 66.82 49.02
VisDoMRAG GPT-4o 44.11 63.28 31.36 44.09 67.22 50.01
Long context Qwen2-VL-7B 8.23 23.10 16.74 9.93 2.46 12.09
VisDoMRAG Qwen2-VL-7B 29.89 59.24 27.98 42.80 39.77 39.94

Across the benchmark VisDoMRAG improves end-to-end QA over long-context, text-only and visual-only baselines by 12-20%, for every LLM we tried (GPT-4o, Gemini 1.5 Flash, Qwen2-VL-7B). The biggest jump is for the small open model: Qwen2-VL goes from 12.09 with the whole collection in context to 39.94 with VisDoMRAG. On the retrieval side, ColQwen2 finds the right source document 96.94% of the time at k=5 versus 92.40% for BGE-1.5, and on SlideVQA the dense text retrievers collapse below 1% because slides carry almost no running text. Removing evidence curation, chain-of-thought and consistency prompting drops VisDoMRAG from 50.01 to 45.98.

Try it