A few dozen frames is already tens of thousands of tokens before you have typed a word, and past the model’s capacity it does not fail gently: it starts emitting repetitive text that has nothing to do with the video. Frames2LoRA, which I worked on with Sarvesh Baskar and Dinesh Manocha at UMD, takes the video out of the context entirely. It reads the frames once, writes what it saw into the model’s weights as a small adapter, and from then on every question is answered from text alone.

The idea in one picture

Doc-to-LoRA showed this works for text documents. Video is harder in three ways. The token volume per example is orders of magnitude larger, the compression is cross-modal (visual content has to become perturbations to a language model’s weights), and video varies along frame count and resolution, axes that text does not have. The rest of this post follows the pipe-lighting clip through the method, one stage at a time.

Walkthrough: one CaReBench clip, from frames to an answer with no frames

Under the hood

The whole method is three lines. The encoder produces video-conditioned states, the hypernetwork maps them to an adapter, and the answer model conditions on the prompt and the adapter but never on the video tokens:

Inside a frozen linear layer with weight \(\mathbf{W} \in \mathbb{R}^{d_{\mathrm{out}} \times d_{\mathrm{in}}}\), the generated factors act as an ordinary LoRA update. In the row-vector convention the layer computes \(\mathbf{x}\mathbf{W}^\top\) and the adapter adds a rank-\(R\) term, which is the same as perturbing the weight by \(\Delta\mathbf{W} = s\,\mathbf{B}^\top\mathbf{A}\):

Training is teacher-forced cross-entropy over response tokens. A frozen SmolVLM2 teacher that does see the frames writes captions and summaries offline; the student answer model has to reproduce them from the prompt and the adapter alone, and the gradient flows only into \(\phi\):

Two details in the hypernetwork matter more than they look. First, the Perceiver bottleneck is what makes frame count a free variable: the latent queries produce a fixed-size summary whether \(S\) covers 8 frames or 1,024, which is why a model trained only at 12 frames and 384 px can be run at 1,024 frames and 1024 px. Second, the zero-initialized \(\mathbf{B}\) scale means training starts from the base model’s behavior and learns a perturbation, rather than starting from a random adapter that has to be unlearned. Training data are spans from FineVideo, mixed 60/30/10 across single-scene, adjacent multi-scene and full-video spans, with audio excluded.

The finding I did not expect is that adapters compose in rank space. Split a video into two temporal halves, internalize each independently, and concatenate the two rank-16 adapters along the rank dimension. Nothing in training ever saw a composed adapter, yet on VDC the composed adapter keeps 93.1% of the single-video adapter’s mean token-F1 at 500M (0.206 vs 0.221) and 86.2% at 2.2B (0.211 vs 0.245), and it produces coherent video-level captions rather than text tied to one half.

The rank directions are redundant but not interchangeable. Ranking each rank slice by the product of its factor norms and keeping only the top 8 gives 0.1264 token-F1 on ActivityNet Captions, against 0.1262 for the full rank-16 adapter, while the lowest-scoring single slice lands below the zero-adapter baseline. The ordering is the same in every one of 500 examples (direction R11 always scores highest), which suggests the hypernetwork has learned a fixed coordinate system for its output. Layer-wise removal on the 2.2B model adds that the updates whose removal hurts most sit in the later layers, close to the output logits.

What the numbers say

• Quality holds. On all five captioning benchmarks (ActivityNet Captions, PLM-RDCap, PLM-RCap, VDC, CaReBench), at both 500M and 2.2B, Frames2LoRA is statistically non-inferior and equivalent to video-in-context inference under an LLM judge (Spearman 0.823 with human ratings), recovering 91.9% of the base judge score at 2.2B and 84.2% at 500M. Video QA was never trained on, yet 7 of 8 benchmark-scale pairings pass, and on NExT-QA the adapter beats the base at both scales.
• Queries get cheap. Across a sweep of 8 to 1,024 frames and 224 to 1024 px, query TTFT falls by a geometric mean of 6.7x at 500M and 20.1x at 2.2B (maximum 79.1x), and answer-time input tokens fall by 150x and 302x on average, reaching 713x and 1,507x.
• It survives where in-context inference does not. Trained at 12 frames, the model stays stable through 1,024 frames and 1024 px (average token-F1 change of -0.012 at 500M). At 1024 px and high frame counts direct inference degenerates into repetitive output and Frames2LoRA leads by +0.12 to +0.13 token-F1.

Try it

• Paper page on this site: /papers/frames2lora/
• arXiv: 2606.04351 (v1 was titled Video2LoRA)
• Code: github.com/frames2lora/Frames2LoRA
• Checkpoints: Frames2LoRA-SmolVLM-ckpts on Hugging Face
• Project page with the qualitative-example explorer: frames2lora.github.io
• The audio-visual follow-up: Omni2LoRA

Software-engineering agents improved by double digits once thousands of real repositories were packaged with reproducible builds and tests. Security agents never got that corpus, because a functional bug is easy to verify (a test fails) while a security weakness has to do the opposite: stay latent under the existing tests and surface only under adversarial input. Existing runnable vulnerability datasets are mined from disclosed CVEs, so they grow at the rate humans find and publish bugs. CyberForge is our attempt to manufacture that data instead. I worked on it with Amine Lbath and colleagues during a NIST PREP fellowship in spring 2026, with Dinesh Manocha at UMD; Amine and I share first authorship. This post traces a single guetzli instance from the released corpus, guetzli/vulnerability_FZ_24, through every stage of the pipeline.

The idea in one picture

The SWE-Smith family of bug-injection pipelines accepts an edit once a unit test fails. CyberForge flips the criterion: an edit is accepted only if every unit test still passes and a proof-of-vulnerability input (PoV) crashes the injected build but not the clean one. Neither half means anything alone, so the oracle checks the injection and the PoV jointly.

Walkthrough: one guetzli bug, from injection to training signal

Everything below is the actual instance guetzli/vulnerability_FZ_24, produced by the fuzzer-guided pipeline ("producer": "fuzz_poc_guided") and labeled CWE-125, an out-of-bounds read.

@@ -398,7 +398,7 @@ bool ProcessAPP(const uint8_t* data, size_t* pos, ...)
   VERIFY_LEN(2);
   size_t marker_len = ReadUint16(data, pos);
   VERIFY_INPUT(marker_len, 2, 65535, MARKER_LEN);
-  VERIFY_LEN(marker_len - 2);
+
   // Save the marker type together with the app data.
   std::string app_str(reinterpret_cast<const char*>(
       &data[*pos - 3]), marker_len + 1);

==432==ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 65531 at 0x6fc93620b1f8 thread T0
    #0 ProcessAPP jpeg_data_reader.cc:403:15
SUMMARY: AddressSanitizer: heap-buffer-overflow
    jpeg_data_reader.cc:403:15 in guetzli::ProcessAPP

Under the hood

The admission predicate. Both pipelines end at the same criterion. Written out, an instance \((\delta, x)\) is admitted when

Condition 1 says the weakness survives normal execution, which is what real weaknesses that pass code review and production testing do. Condition 2 is a differential test under identical input, and it is what makes the pair meaningful: a crash on both builds is a pre-existing bug, a crash on neither is an injection nobody can reach. The verifier’s check on sanitizer type and location, folded into \(\mathrm{crash}\) above, rules out spurious failures.

Yield. CyberForge made 16,172 injection attempts and 1,034 passed validation: 643 from the fuzzer-guided pipeline and 391 from the agentic one. Writing a plausible injection is easy; producing one the oracle accepts is the hard part. In the Pipeline 2 ablation, a naive single-pass agent compiles and passes unit tests on 68.2% of attempts and passes validation on 0%; taint analysis alone reaches 2.8% validated, retry loops alone 3.5%, and the full workflow lifts plausible injections to 77.6% and validated yield to 7.5%. Pipeline 1 shows the same from the other side: a post-hoc 90-second libFuzzer replay over a seeded corpus raises yield with the injection stage untouched. The pipelines fail at different stages, Pipeline 1 losing 64.2% of candidates at validation and Pipeline 2 58.8% at injection, but a PoV that never triggers is the largest single cause in both (44.6% and 33.1%).

Realism. For an edit statistic (functions modified, files touched, hunks, lines changed), let \(F_n\) and \(G_m\) be the empirical distributions over the injected corpus and over the 300 real SEC-bench instances. The two-sample Kolmogorov-Smirnov statistic is

A KS distance has no universal threshold, so the floor is measured between two real corpora, the SEC-bench cve and oss splits. For functions modified, injected-to-real is 0.165 against a real-to-real floor of 0.190; files touched and hunk count give 0.123 and 0.243 against 0.065 and 0.205. The edits are small and local in the way CVE patches are: 1,025 of 1,034 instances change one file, 944 confine the change to one hunk, and the median edit is 2 lines. The guetzli instance, one deleted line, is typical.

Training. Over the 1,034 instances the teachers yielded 1,194 accepted trajectories from GPT-5.4-mini and 880 from Gemma 4 31B. Raw trajectories are recorded in the teacher’s environment, so before fine-tuning they are aligned to the SEC-bench harness (secb repro / secb build, PoV paths remapped to /testcase, rg rewritten to grep), linearized to one command per assistant turn, and compressed with a sliding window over old observations so the turn that writes the patch is never truncated. Every student uses the same recipe: LoRA adapters (\(r = 32\), \(\alpha = 64\), dropout 0.05) on the attention and MLP projections with the base frozen, learning rate \(1 \times 10^{-4}\) with a cosine schedule, batch size 1 with gradient accumulation 2, three epochs, BFloat16, one H200. Holding the 12B student and Gemma teacher fixed, SEC-bench rises from 3.6% to 12.1% to 16.0% as the trajectory corpus doubles twice; at 220 trajectories the student scores below its own base, so a corpus too small to teach the workflow is worse than none.

What the numbers say

All six student-teacher configurations improve on SEC-bench, by 3.3 to 14.7 points; the self-distilled students taught by Gemma 4 31B gain too (31B: 64.7%). The corpus is entirely C/C++, yet every configuration also improves on PatchEval’s Go, JavaScript, and Python CVEs, and the 31B student passes its teacher there. What changed inside the agent explains the scores: the 12B base completes an edit-then-verify cycle on 20.7% of instances, and after fine-tuning on 82.7%.

Try it

• Paper page on this site: /papers/cyberforge/
• arXiv: 2608.06471
• Code: github.com/Cyb3rForge/CyberForge
• Data: cyberforge-projects on Hugging Face, one archive per project with the diff, metadata, sanitizer report, and PoV files for every instance, including guetzli/vulnerability_FZ_24
• Project page: cyb3rforge.github.io
• UMD CS wrote about the project: When AI Goes on Defense

Both failures have the same root. The agent reasons about ambiguity in the space of words: it is prompted to write a clarifying question as free text, so it has no explicit picture of which argument is missing or whether the tool’s own schema already pins it down. In our ACL 2026 Findings paper, Structured Uncertainty Guided Clarification for LLM Agents, we with collaborators at Adobe Research and UMD move the uncertainty into the space of tool parameters and their domains. Once it lives there, “should I ask?” and “what should I ask?” become quantities the agent can compute.

The idea in one picture

Walkthrough: calling Alex, then texting Maya

The walkthrough traces both requests through SAGE-Agent (Structured Argument Uncertainty guided Elicitation Agent). The belief values are worked by hand from the paper’s Equation 2 with the settings we used in the experiments, a redundancy penalty of \(\lambda = 0.5\) and a stopping coefficient of \(\alpha = 0.1\).

Under the hood

The clarification loop is a sequential decision problem: at each turn the agent chooses between executing its best candidate and asking a question, with the user’s true intent hidden. The paper frames this as a POMDP whose observations are user answers and whose belief state is a distribution over structured tool calls, and uses a Bayesian value-of-information objective to choose questions.

Belief. With a uniform prior over tools and conditional independence across parameters, the belief in a candidate factors over its parameters:

An answer \(r_t\) to question \(q_t\) updates each targeted domain by intersection, \(\mathcal{D}_\theta(t+1) = \mathcal{D}_\theta(t) \cap \mathrm{Update}(\theta, r_t, q_t)\), and the beliefs are renormalized. Answers can be explicit values, cross-parameter dependencies, or exclusions such as “not business class”; all three are domain constraints, which is what keeps the update exact.

Value of a question. The Expected Value of Perfect Information of \(q\) is the expected gain in best-candidate certainty if the aspects it targets were resolved perfectly:

In practice we simulate the resolution directly: for each candidate, multiply \(\pi_c(t)\) by \(\lvert \mathcal{D}_a \rvert\) for every targeted aspect it leaves unspecified, then take the expected maximum. EVPI is non-negative, has diminishing returns over question sequences, and goes to zero as the belief concentrates, which is exactly what happened to “Which Alex?” in step 4.

Cost, selection, stopping. Asking about the same aspect twice is discouraged by a redundancy cost, and the agent picks the question with the best net score:

The agent also executes immediately when \(\max_c \pi_c(t) \geq \tau_{\mathrm{exec}}\), and stops after \(n_s\) steps. Because the stopping bar scales with the current best belief, the loop ends either because the belief has become confident or because no remaining question is worth its cost. In the experiments we set \(\lambda = 0.5\), \(\alpha = 0.1\), and \(\epsilon = 10^{-4}\).

The same belief as a training signal. The certainty \(\max_c \pi_c(t)\) also makes a reward. In GRPO training on 9K When2Call examples, we multiply the action-classification reward by \(\mathrm{Cert}(a_t)\), which is \(\max_c \pi_c(t)\) for a tool call, \(1 - \max_c \pi_c(t)\) for a clarifying question, and 1 otherwise, so confident correct calls get full payoff, low-certainty calls are penalized, and asking is rewarded only when uncertainty is high, with no critic to judge question quality.

Where the evaluation comes from. We built ClarifyBench to test this in multi-turn use: 716 tasks across document processing, vehicle control, stock trading, travel, and file systems, over 92 tools, with an LLM user simulator that holds the true intent, answers questions, and issues follow-up requests. Queries are split into explicit, ambiguous, and infeasible (which should be declined), and every generated query was checked by two annotators.

What the numbers say

Coverage counts tool calls that fully match the ground truth; TMR and PMR are tool and parameter match rates; #Q is the mean number of clarifying questions per task. Across baselines and both base models (GPT-4o and Qwen2.5-14B-Instruct), SAGE-Agent reaches 7-39% higher coverage on ambiguous tasks while asking 1.5-2.7x fewer questions. The heuristic row triggers questions on <UNK> alone without EVPI scoring, and gives up 1-3 points across metrics while asking 0.2-0.4 more questions, which is the value of steps 4 and 5 above. Raising \(\lambda\) from 0 to 0.5 cuts questions by 18.1% (ambiguous), 26.6% (explicit), and 24.2% (infeasible) with Coverage, TMR, and PMR within 3%, so the pruned questions were redundant. On the training side, certainty-weighted GRPO lifts When2Call accuracy from 36.5% to 65.2% for Qwen2.5-3B and from 36.7% to 62.9% for Qwen2.5-7B; the 3B model with our reward beats the 7B model with the standard reward (45.1%).

Try it

• Paper page: /papers/sage-clarification/
• arXiv: 2511.08798
• Code and benchmark: github.com/MananSuri27/ClarifyBench
• ACL Anthology: 2026.findings-acl.2028

Text LLMs got a partial fix for this years ago: citations. Make the model point at its source and a human can verify the claim in seconds. Charts never had an equivalent. Our ACL 2025 paper, ChartLens: Fine-grained Visual Attribution in Charts, with Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan Rossi, and Dinesh Manocha, gives charts their citations: the specific bars, points, or pie sectors that an answer rests on.

The idea in one picture

Walkthrough: the documents-per-shipment chart

The running example is Figure 1 of the paper, a PlotQA-style chart of the number of documents required per shipment to import goods, for six countries in 2005, 2006, and 2007. I redraw it below so each stage can be shown on the same bars.

Walkthrough: attributing "Ans: 3" to the bars that support it

1. A chart, a question, and someone's answer

The input (chart \(c\) plus response \(v\)) is the image and a question-answer pair. The answer could come from any system, a chart QA model or a person; ChartLens does not regenerate it. The job is to return the set of chart regions that justify “3”, and it has to be relevant (every region matters), complete (nothing needed is missing), and precise (nothing extra).

2. Find every bar

Before any language model is involved, classical vision finds the candidates. We binarize the image with Otsu thresholding on both RGB and HSV, invert if the background is dark, extract contours, and split each contour by unique pixel value so a group of touching bars separates into three. Solidity and area thresholds drop stray shapes. Because these heuristics are brittle on low-contrast charts and can pick up grid lines or labels, we sample \(n\) points inside each candidate and hand them to the Segment Anything Model (SAM), which returns a tight mask per bar and only weak masks for grid lines and text, so those fall away. For this chart the result is eighteen masks.

3. Stamp a label on each element

Each mask gets an alphanumeric label drawn directly on the image, following set-of-marks (SoM) prompting. Reading left to right, Algeria’s three bars become 1, 2, 3, Angola’s 4, 5, 6, and so on through Australia’s 16, 17, 18. From here on the model never has to name a position in pixels; it names a label, and we already know which mask that label points to.

Algeria 2005Algeria 2006Algeria 2007...Australia 2007→123...18

4. The model checks the answer against the chart

The marked image goes to the MLLM (GPT-4o in our experiments) with a prompt that explains what chart attribution is, gives a few textual examples of question, answer, and attribution, and asks for chain-of-thought reasoning in two steps. Step one is validation: is the answer consistent with the chart? For our question the model has to read the six 2005 bars, take their mean, and count how many exceed it.

1: 94: 87: 510: 613: 616: 7→mean 6.83→9, 8, 7 above→count 3 ✓

5. The model names the marks that support it

Step two is attribution: which labeled elements support the answer? The model returns 1, 4, and 16, the 2005 bars for Algeria, Angola, and Australia. We map those labels back to their SAM masks and highlight them. The reader’s verification problem has shrunk from “read this chart” to “count these three bars”, and the count matches the answer.

Ans: 3→MLLM→1416→3 bars, consistent

6. The same pipeline on a wrong answer

Now the second question from Figure 1: “In which year did Armenia have the maximum number of documents required per shipment to import goods?” with the answer 2006. Armenia’s bars read 6, 6, 8, so the maximum is 2007, and the validation step flags the response as inconsistent. The attribution still grounds the answer to the bar it depends on, Armenia’s 2006 bar (mark 14), and one glance shows it is not the tallest of the three. That is what a chart-response misalignment looks like when it is made visible.

Ans: 2006→MLLM→inconsistent14→15 is taller

Under the hood

The task is post-hoc fine-grained visual attribution for charts. Given a chart and a response, produce

where each \(a_i\) is a distinct chart element that supports \(v\), subject to relevance (each \(a_i\) bears on \(v\)), completeness (\(\mathcal{A}_{c,v}\) covers all the evidence needed to justify \(v\)), and precision (no irrelevant parts of the chart). The whole method is a mapping

and the design question is how to make an MLLM implement \(f\) when it cannot reliably output coordinates. ChartLens answers by turning region selection into label selection: the candidate set of elements is computed by segmentation, and the model chooses a subset of labels.

Mark generation. Bars follow the pipeline in step 2. Pie charts take the largest contour of the binarized image, fit its minimum enclosing circle, unroll the pie along the radial axis into a strip, and detect complete edges in that strip as sector boundaries, which map back to slices. Both then pass through SAM with sampled point prompts (facebook/sam-vit-large). Lines are thin, overlapping, and intersecting, so contour heuristics do not apply; we use LineFormer, a transformer-based line extractor, and divide each recovered line into equally spaced segments along its horizontal extent so that a point on a line has a mark to refer to.

Attribution prompt. The prompt has three parts: a description of chart attribution, few-shot textual examples of question-answer pairs with their attributions, and an instruction to reason step by step through validation (is the QA pair consistent with the chart?) and then attribution (which labeled elements support the answer?). For line charts the model returns pairs of marked points between which the attributed span lies; those pairs are treated as bounding-box corners.

Scoring. For bars and sectors, a predicted region counts as a match if it overlaps a ground-truth region at \(\text{IoU} \geq 0.9\). With \(D\) the matched detections and \(G\) the ground truth,

Lines are scored by detection rate (the fraction of ground-truth points covered, a recall-like measure) and by the percentage of the chart’s area the attribution covers, because a method can reach high detection by painting most of the chart.

ChartVA-Eval. To measure any of this we built a benchmark of 1,244 queries over three subsets. ChartVA-AITQA renders synthetic charts from airline SEC-filing tables (301 queries, 203 bar and 98 line charts, one attribution each). ChartVA-PlotQA uses synthetic scientific charts from World Bank Open Data, Open Government Data, and the Global Terrorism Database (595 queries, 396 bar and 199 line charts, 2.4 attributions on average, up to 12). ChartVA-ChartQA uses real charts from Statista, Pew Research, Our World in Data, and OECD, with pie charts oversampled (348 queries: 121 bar, 109 pie, 118 line). For the last two, GPT-4o drafted attributions from the underlying tables with template-specific prompts, and three annotators verified relevance and completeness with Cohen’s kappa of 0.89 and 0.84.

What the numbers say

Baselines are zero-shot GPT-4o bounding-box prompting, Kosmos-2, and LISA; ChartLens uses GPT-4o as its MLLM.

On line charts ChartLens covers 59.14%, 51.84%, and 77.8% of ground-truth points on the three subsets while flagging 1.25%, 9.98%, and 5.34% of the chart area; LISA and Kosmos-2 reach high detection mainly by covering 27% to 63% of the chart, so ChartLens uses 3 to 50 times less area. Across chart types this is a 26-66% improvement in fine-grained attribution over the baselines. Qualitatively, GPT-4o tries to be specific but cannot localize through text coordinates, and LISA and Kosmos-2 return generic components such as the whole pie regardless of the question.

Try it

• Paper page: /papers/chartlens/
• arXiv: 2505.19360
• Code and the ChartVA-Eval benchmark: github.com/MananSuri27/ChartLens
• ACL Anthology: 2025.acl-long.1094

Most document QA demos skip all of that by starting from one PDF. Our NAACL 2025 paper, VisDoM, is about the version of the problem that analysts, scientists and lawyers actually have: a question over a collection of documents where the evidence is usually visual. We built a benchmark for it (VisDoMBench) and a retrieval-augmented method that answers it (VisDoMRAG). This post traces that Ubuntu question through VisDoMRAG one stage at a time.

The idea in one picture

A retrieval-augmented generation (RAG) system for PDFs has to pick a modality. Retrieve text and the model reads tables as flattened OCR strings. Retrieve page images and the model sees the table as pixels, which is better for tables and charts but leaves it alone with whatever it misreads. Our answer is to do both, in two independent pipelines, and then make a language model check the two reasoning chains against each other before committing.

Walkthrough: one question, 129 pages

Walkthrough: "Which model performs best on the Ubuntu dataset for text lengths between 60 and 90 words?"

1. The pile of PDFs and the question

This is a PaperTab-style query from VisDoMBench. The document that answers it is “Knowledge Enhanced Hybrid Neural Network for Text Matching” (the KEHNN paper), and the evidence is its Table (b), the Ubuntu dataset results. Every query in the benchmark comes with distractor documents, so the collection spans roughly 50 to 200 pages and the system has to locate the document before it can locate the table.

2. Index the collection twice, in parallel

VisDoMRAG never chooses between pixels and text; it builds both indexes. The visual branch renders every page as an image and embeds it with a late-interaction visual retriever (ColQwen2; ColPali also works), with no OCR at all. The textual branch OCRs every page with PyTesseract, splits the text into 3,000-character chunks with 10% overlap (recursive split), keeps the source document and page number as metadata, and embeds each chunk with BGE-1.5. Both indexes are built once per collection.

3. Retrieve in both modalities

The same question is scored against both indexes. The visual retriever returns the top-5 pages, and the page of the KEHNN paper that carries Table (b) is among them; so are pages from distractor papers, which is what makes the later curation step necessary. The textual retriever returns the top-7 chunks, one of which is the OCR of that same table, flattened into a string like “KEHNN 0.724 0.774 0.785 0.791”. We picked k=5 and k=7 as the smallest windows whose retrieved evidence overlapped the ground-truth evidence at ANLCS 0.7 or better (Figure 3 of the paper).

4. Each branch curates evidence, reasons, then answers

Neither branch is asked for an answer straight away. The LLM is prompted three times: first to pull out and write down the evidence that matters (evidence curation), then to link that evidence into an explicit argument (chain-of-thought reasoning), and only then to answer in the format the question type calls for (answer generation). In the visual branch the multimodal LLM reads the [60, 90) column off the page image: LSTM 0.732, MV-LSTM 0.725, KEHNN 0.785. In the textual branch the LLM has to recover the same column from the flattened OCR row. Curation is what stops the distractor pages and chunks from leaking into the reasoning.

5. Fuse the two chains with a consistency check

A final LLM call receives the curated evidence, the reasoning chain and the answer from both branches and is asked whether the two chains are consistent (modality fusion). When they agree, as they do for the Ubuntu question, the answer is confirmed with two independent lines of evidence behind it. When they conflict, the LLM has to re-examine the evidence and reconcile the difference. The paper’s PaperTab example (Figure 5) shows the conflict case: asked for the size of the StackEx keyphrase dataset, visual RAG latched onto 57.5% and textual RAG onto 298k, both wrong numbers from the right table; the fused reasoning summed the splits and answered around 330k.

6. The final answer, with its evidence attached

query→5 pages7 chunks→visual chaintextual chain→fusion→KEHNN

The curated evidence survives to the output, so the answer arrives with the table it came from. Rebuilt from Table (b) of the KEHNN paper (the grounding context in Figure 1):

Length	[0, 30)	[30, 60)	[60, 90)	[90, ∞)
#Pair	253578	207772	33618	5032
LSTM	0.707	0.748	0.732	0.718
MV-LSTM	0.726	0.752	0.725	0.694
KEHNN	0.724	0.774	0.785	0.791

Final answer: KEHNN. Because both branches wrote their evidence down before answering, the response is verifiable: you can point at the cell that produced it (the paper calls this implicit context attribution).

Under the hood

Retrieval. The visual retriever scores a page by late interaction (MaxSim): every query token embedding is matched to its most similar patch embedding of the page, and the similarities are summed.

The textual retriever is a dense bi-encoder (BGE-1.5), scoring chunks by embedding similarity to give \(\mathcal{C}_q\), the top-\(k_t\) chunks. We benchmarked BM25, MiniLM, MPNet and BGE-1.5 on the text side and ColPali and ColQwen2 on the visual side; ColQwen2 and BGE-1.5 were the best of each and are what the end-to-end numbers use.

Per-branch prompting. Each branch is the same three-call chain, applied to its own context:

The curation call is the one that does the multi-document work: it must isolate the paragraphs, table rows or figure details that bear on \(q\) and verbalize them in a structured form, dropping the distractor content that retrieval let through.

Consistency-constrained fusion. The final answer is one more LLM call over everything both branches produced, with the instruction to judge whether \(R_v\) and \(R_t\) are consistent and to reconcile them if not:

This is a late-fusion design, in the spirit of self-consistency over chains of thought: each modality is processed independently and the chains are compared afterwards. The early-fusion alternative, appending the OCR text of the visually retrieved pages to the image prompt, scores 43.63 average with GPT-4o against 50.01 for late fusion.

The benchmark. VisDoMBench re-purposes five datasets that have public source documents and grounded evidence, de-duplicates questions across splits, drops trivial ones, and augments each question with distractor documents. Ambiguous questions (common in PaperTab and SciGraphQA) are rewritten by GPT-4o into more specific variants and a human annotator picks one, keeps the original, or discards the item, so that exactly one document answers each question.

Answers are scored with word-overlap F1 (the UDA variant for PaperTab, which handles binary and short-text answers). A retriever is credited with identifying the source document when at least \(\lceil k/2 \rceil\) of its top-\(k\) results come from the ground-truth document.

What the numbers say

Across the benchmark VisDoMRAG improves end-to-end QA over long-context, text-only and visual-only baselines by 12-20%, for every LLM we tried (GPT-4o, Gemini 1.5 Flash, Qwen2-VL-7B). The biggest jump is for the small open model: Qwen2-VL goes from 12.09 with the whole collection in context to 39.94 with VisDoMRAG. On the retrieval side, ColQwen2 finds the right source document 96.94% of the time at k=5 versus 92.40% for BGE-1.5, and on SlideVQA the dense text retrievers collapse below 1% because slides carry almost no running text. Removing evidence curation, chain-of-thought and consistency prompting drops VisDoMRAG from 50.01 to 45.98.

Try it

• Paper page on this site: /papers/visdom/
• arXiv: 2412.10704; ACL Anthology: 2025.naacl-long.310
• Code and the VisDoMBench splits: github.com/MananSuri27/VisDoM

I have always been interested in learning different languages, though the only French the Duolingo owl has taught me is Je m’appelle Manan. My short stints at learning a third language helped me realise that vocabulary is not sufficient: contexts, semantics and syntactic features are also important to truly grasp meaning from a language. So when I started doing NLP tasks, I felt perplexed about how something like a bag-of-words, which considers each word in a very independent light, could really be effective. I got answers to this question when I learnt what word embeddings are and how they work. In this article, I’m going to talk about word embeddings, specifically the Word2Vec model, and how to take advantage of it using the easy-to-use Gensim library.

Word embeddings

The emergence of language was a pivotal moment in the evolution of humanity. Although all species have their ways of communicating, we as humans are unique in having mastered cognitive language communication. So while I know that “rat” refers to a small hairy rodent, my dog or my computer (at least in essence) doesn’t know that.

Therefore, any task aimed at processing language must first begin with how to represent words.

A preliminary method is a “bag of words” model which encodes words using a one-hot scheme. If our dataset contains the sentences:

“I like the new movie!”, “I love the weather.”

Then we can have a vector representation of the words as:

I       [1,0,0,0,0,0,0]
like    [0,1,0,0,0,0,0]
the     [0,0,1,0,0,0,0]
new     [0,0,0,1,0,0,0]
movie   [0,0,0,0,1,0,0]
love    [0,0,0,0,0,1,0]
weather [0,0,0,0,0,0,1]

The sentences will then be represented as [1,1,1,1,1,0,0] and [1,0,1,0,0,1,1].

However, as you might have noticed, this representation is not very effective at showing the semantic and syntactic relationships between words. They are encoded as individual bits in a vector space, and there is no way you can tell that the words “love” and “like” have a similar connotation.

This is where word embeddings come in. Word embeddings are representations where contexts and similarities are captured by encoding in a vector space: similar words have similar representations. We’re going to discuss Word2Vec, which is an effective word embedding technique.

Word2Vec

Word2Vec creates a representation of each word in our vocabulary as a vector. Words used in similar contexts, or having semantic relationships, are captured through their closeness in the vector space. Effectively, similar words will have similar word vectors! Word2Vec was created, patented, and published in 2013 by a team of researchers led by Tomas Mikolov at Google.

Let us consider a classic example: “king”, “queen”, “man”, “girl”, “prince”.

In a hypothetical world, vectors could define the weight of each criterion (for example royalty, masculinity, femininity, age) for each of the words in our vocabulary. What we then observe is:

• As expected, “king”, “queen”, “prince” have similar scores for “royalty”, and “girl”, “queen” have similar scores for “femininity”.
• An operation that removes “man” from “king” would yield a vector very close to “queen” (“king” − “man” = “queen”).
• Vectors “king” and “prince” have the same characteristics except for age, telling us how they might be semantically related to each other.

Word2Vec forms word embeddings that work in a similar fashion, except that the criteria used for each word are not clearly determinable. What matters to us is the semantic and syntactic relations between words, which can still be determined by our model without explicitly defining features for units of the vector.

Word2Vec has also been shown to identify relations like country–capital over larger datasets, showing how powerful word embeddings can be. Embeddings generated by Word2Vec can further be used in NLP tasks, such as feeding them into a CNN to classify text.

Model architecture

Word2Vec is essentially a shallow, 2-layer neural network.

• The input contains all the documents/texts in our training set. For the network to process these texts, they are represented as one-hot encodings of the words.
• The number of neurons in the hidden layer equals the length of the embedding we want. That is, if we want all our words to be vectors of length 300, then the hidden layer will contain 300 neurons.

• The output layer contains probabilities for a target word (given an input to the model, what word is expected).
• At the end of the training process, the hidden weights are treated as the word embedding. Intuitively, this can be thought of as each word having a set of n weights (300 in the example above) “weighing” their different characteristics, the analogy we used earlier.

There are two ways in which we can develop these embeddings.

1. Continuous Bag-Of-Words (CBOW). CBOW predicts the target word based on its surrounding words. For example, consider the sentence “The cake was chocolate flavoured”. The model will iterate over this sentence for different target words, such as “The __ was chocolate flavoured” being the input and “cake” being the target word. CBOW thus smoothes over the distribution of the information, as it treats the entire context as one observation. CBOW is faster than skip-gram and works well with frequent words.

2. Skip-gram. Skip-gram works in the exact opposite way to CBOW. Here we take an input word and expect the model to tell us what words it is expected to be surrounded by. Taking the same example, with “cake” we would expect the model to give us “The”, “was”, “chocolate”, “flavoured”. The statistical interpretation of this is that we treat each context–target pair as a new observation. Skip-gram works well with small datasets and can better represent less frequent words.

Using Gensim to train our own embeddings

We can easily train Word2Vec embeddings using Gensim, which is “a free open-source Python library for representing documents as semantic vectors, as efficiently (computer-wise) and painlessly (human-wise) as possible.”

The dataset I used for this demo is the Coronavirus tweets NLP dataset from Kaggle. I am omitting the parts involving loading the dataset and preprocessing the text, but you can check out the complete implementation in this Colab notebook. I preferred this over larger datasets like IMDB because many real applications involve similar-sized datasets, so it makes for a better representation of the model’s average performance.

1. Training the embeddings

We import Word2Vec from gensim.models. Each input to the model must be a list of tokens, so we generate the input by calling split() on each line in our corpus of texts.

from gensim.models import Word2Vec

sentences = [line.split() for line in texts]

w2v = Word2Vec(sentences, vector_size=100, window=5, workers=4, epochs=10, min_count=5)

print(sentences[20:25])
# [['with', 'nations', 'inficted', 'with', 'covid', 'the', 'world', 'must', 'not', 'play', 'fair', ...

We then set up the model and specify different parameters. Briefly, what they mean:

• vector_size (called size in Gensim 3) is the size of the word embedding it will output.
• window is the maximum distance between the current and predicted word within a sentence.
• min_count sets a minimum frequency for a word to be part of the model; all words with count less than min_count are ignored.
• workers is the number of worker threads used to train the model. This can be adjusted to the number of cores your system has. In simple words, it is the parallelism while training.
• epochs (called iter in Gensim 3) is the number of passes over the corpus during training.

2. Using the Word2Vec model

Finding the vocabulary of the model can be useful in several general applications, and in this case it gives us a list of words we can try with the other functions.

words = list(w2v.wv.index_to_key)   # w2v.wv.vocab in Gensim 3
print(words)
# ['phil', 'advice', 'talk', 'to', 'your', 'neighbours', 'family', 'exchange', 'phone', 'numbers', ...

Finding the embedding of a given word is useful when we’re trying to represent sentences as a collection of word embeddings, like when building a weight matrix for the embedding layer of a network. I include this so it can help your intuition of what a word vector looks like.

print(w2v.wv['computer'])
# [ 1.57469660e-01  1.40157074e-01 -3.25907797e-01 -6.61702231e-02
#   3.14891905e-01  6.28795177e-02 -4.47840840e-02  4.59685735e-02
#   ...  (100 values)

As you can see, it is not possible to make sense of what these individual values mean, unlike the completely hypothetical example I gave above.

We can also find the similarity between given words (the cosine similarity between their vectors). Here we compare “vladimir” with “putin” and with “modi”, and a stark distinction exists.

w2v.wv.similarity('vladimir', 'putin')
# 0.81842446

w2v.wv.similarity('vladimir', 'modi')
# 0.6622772

With Gensim we can also find the most similar words to a given word. This particularly shows the contextualising power of the model. Look at words similar to “covid”: we get “coronavirus”, “virus”, “corona”, “disease” as the top words. When we try “india”, we get a list of words that are also countries! When we try a verb such as “pay”, we get other forms of the same verb, “paid”, “paying”, and associated terms like “wages” and “bills”. This is exciting considering our vocabulary is not very large and the dataset covers a very specific situation.

print(w2v.wv.most_similar('pay'))
# [('paying', 0.702), ('paid', 0.686), ('wages', 0.651), ('bills', ...), ...]

print(w2v.wv.most_similar('covid'))
# [('coronavirus', 0.603), ('virus', 0.529), ('corona', 0.507), ...]

print(w2v.wv.most_similar('india'))
# [('nigeria', 0.753), ('pakistan', 0.751), ('kenya', 0.683), ...]

Similarly, we can use the same function to find analogies of the form if x : y, then z : ?. We enter the known relation in the positive parameter and the term whose analogue we want in the negative parameter. Here our model seems to have learnt something about nationalities: if “russian” → “russia”, then “arab” → “saudi”, “arabia” (taking the first two terms, because the model did not treat multi-word phrases as single tokens).

print(w2v.wv.most_similar(positive=['russian', 'russia'], negative=['arab']))
# [('saudi', 0.798), ('arabia', 0.774), ('putin', 0.735), ...]

There is also a method that works like an “odd one out” puzzle. Here the model identifies “grocery” as different from “covid” and “coronavirus”.

w2v.wv.doesnt_match(['grocery', 'covid', 'coronavirus'])
# 'grocery'

3. Visualising word embeddings

Word2Vec embeddings are usually of size 100 or 300, and it is not practical to visualise a 100- or 300-dimensional space meaningfully. I used a snippet from Stanford’s CS224N course site, which lets you either pass a list of words or a number of random samples to display. In either case, it uses PCA to reduce the dimensionality and plots the words on a 2-dimensional plane. The actual axis values hold no significance; what we can see is that similar vectors are densely located with respect to each other.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA

def display_pca_scatterplot(model, words=None, sample=0):
    if words is None:
        words = list(model.wv.index_to_key) if sample == 0 else \
                np.random.choice(list(model.wv.index_to_key), sample, replace=False)
    word_vectors = np.array([model.wv[w] for w in words])
    twodim = PCA().fit_transform(word_vectors)[:, :2]

    plt.figure(figsize=(6, 6))
    plt.scatter(twodim[:, 0], twodim[:, 1], edgecolors='k', c='r')
    for word, (x, y) in zip(words, twodim):
        plt.text(x + 0.05, y + 0.05, word)

display_pca_scatterplot(w2v, ['coronavirus', 'covid', 'virus', 'corona', 'disease',
                              'saudiarabia', 'doctor', 'hospital', 'pakistan', 'kenya',
                              'pay', 'paying', 'paid', 'wages', 'raise', 'bills', 'rent', 'charge'])

On the graph, you can see how “coronavirus”, “covid”, “virus” form one group, separate from the others, while “paying”, “paid”, “bills”, “wages” are in another group altogether. Similarly, the countries “saudiarabia”, “kenya”, “pakistan” form one very dense cluster.

4. Saving models, and using pre-trained models

Gensim ships with several pre-trained models in the gensim-data repository. We can import the downloader from the Gensim library and print the list of available pre-trained models. This also includes models like GloVe and fastText, not only Word2Vec.

import gensim.downloader

print(list(gensim.downloader.info()['models'].keys()))
# ['fasttext-wiki-news-subwords-300', 'conceptnet-numberbatch-17-06-300',
#  'word2vec-ruscorpora-300', 'word2vec-google-news-300', 'glove-wiki-gigaword-50', ...]

Here we use word2vec-google-news-300 (trained on Google News, with 300-dimensional vectors) and find words similar to “twitter”.

google_news = gensim.downloader.load('word2vec-google-news-300')
google_news.most_similar('twitter')
# [('Twitter', 0.891), ('Twitter.com', 0.754), ('tweet', 0.743), ('tweeting', 0.716),
#  ('tweeted', 0.714), ('facebook', 0.699), ('tweets', 0.697), ...]

We can save our trained models, load them again, and even continue training them.

w2v.save("word2vec.model")

model = Word2Vec.load("word2vec.model")
model.train([["hello", "world"]], total_examples=1, epochs=1)

Summary

• Word embeddings are a better way to represent natural language than a skeletal bag-of-words. They capture the semantic and syntactic relationships present in text.
• Word2Vec represents each word in our vocabulary as a vector, so that similar words have similar word vectors.
• Word2Vec embeddings can be trained in two ways: CBOW predicts the target word from its surrounding words; skip-gram does the opposite, predicting surrounding words for a given input word.
• We can easily train Word2Vec embeddings using Gensim, a free open-source Python library, on our own corpus, or use pre-trained embeddings.
• Gensim provides functions for working with embeddings, including finding similar words, computing similarities, and solving analogies.
• The Gensim downloader gives easy access to embeddings trained on large datasets like Google News. We can save our trained models and continue training them later.

Code: Colab notebook

Postscript (2026): what Word2Vec got right that we forgot in the LLM era

Four years after writing this, I spend most of my time on LLM agents, and it is easy to treat Word2Vec as a museum piece. It isn’t. A few things it got right that are worth remembering:

• The distributional hypothesis is still the whole game. “You shall know a word by the company it keeps” is exactly what a transformer’s next-token objective optimises, just with a much bigger context window and a much bigger model. Word2Vec was the cleanest possible demonstration that predicting neighbours is enough to learn meaning. Everything since is scaling that same idea.
• Static embeddings are underrated. A lookup table is the cheapest model there is: no forward pass, no GPU, no latency. For retrieval pre-filtering, deduplication, vocabulary analysis, or anything that runs millions of times, a good static embedding is often the right tool, and modern “static” models distilled from LLMs bring most of the quality at a tiny fraction of the cost.
• Efficiency was a design goal, not an afterthought. Negative sampling, hierarchical softmax, and subsampling of frequent words exist because Mikolov’s team wanted to train on billions of tokens on a CPU. That mindset, do the simplest thing that scales, is one I keep coming back to when thinking about how agents should hold context without paying for it on every query.
• Probing the geometry is how you build intuition. most_similar, analogies, and doesnt_match are toy tools, but they teach you to ask “what has this model actually learnt?” That habit transfers directly to interpreting, attributing, and trusting what large models produce.

References / further reading

• Word2Vec explained
• The Illustrated Word2Vec by Jay Alammar
• Gensim Word2Vec tutorial on Kaggle
• Word2Vec demo
• Mikolov et al., Efficient Estimation of Word Representations in Vector Space (2013)