Put a bar chart in front of a multimodal LLM and ask in how many countries the number of import documents in 2005 is above the 2005 average across countries. It answers “3” in a full sentence, instantly, and with total confidence. Now ask yourself how you would check that. You would have to read six bars, compute a mean, and count. The model did none of that visibly. If it misread one bar, the text would look exactly the same.

Text LLMs got a partial fix for this years ago: citations. Make the model point at its source and a human can verify the claim in seconds. Charts never had an equivalent. Our ACL 2025 paper, ChartLens: Fine-grained Visual Attribution in Charts, with Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan Rossi, and Dinesh Manocha, gives charts their citations: the specific bars, points, or pie sectors that an answer rests on.

The idea in one picture

Chart QA today Q: how many countries above the 2005 average? Ans: 3 Which bars is that based on? The answer looks fine whether it is right or wrong. Nothing in the output points back at the chart. Chart QA + ChartLens Q: how many countries above the 2005 average? Ans: 3 These three bars. Count them: 3. Attribution = the bars, points, or sectors that support the answer, so a reader can verify it.
Left: a chart answer with nothing pointing back at the chart. Right: the same answer with the three bars that support it highlighted, so verifying it means counting to three.
Key ideaDo not ask the model for pixel coordinates; MLLMs are bad at that. Find the chart's elements first with segmentation, stamp a label on each one, and ask the model which labels support the answer (set-of-marks prompting). Attribution runs after the answer exists, so it is a check on any chart QA system rather than a new one.

Walkthrough: the documents-per-shipment chart

The running example is Figure 1 of the paper, a PlotQA-style chart of the number of documents required per shipment to import goods, for six countries in 2005, 2006, and 2007. I redraw it below so each stage can be shown on the same bars.

Walkthrough: attributing "Ans: 3" to the bars that support it

1. A chart, a question, and someone's answer

Number of documents required per shipment to import goods 2005 2006 2007 0 2 4 6 8 Documents required Algeria Angola Antigua and Barbuda Argentina Armenia Australia Question In how many countries, is the number of documents required per shipment to import goods in 2005 greater than the average number of documents required per shipment to import goods in 2005 taken over all countries? Ans: 3

The input (chart \(c\) plus response \(v\)) is the image and a question-answer pair. The answer could come from any system, a chart QA model or a person; ChartLens does not regenerate it. The job is to return the set of chart regions that justify “3”, and it has to be relevant (every region matters), complete (nothing needed is missing), and precise (nothing extra).

2. Find every bar

Number of documents required per shipment to import goods 2005 2006 2007 0 2 4 6 8 Documents required Algeria Angola Antigua and Barbuda Argentina Armenia Australia Otsu thresholdRGB + HSV, invert if dark Contourssplit by unique pixel value Filtersolidity + area thresholds SAMn points per candidate as prompts Output: one tight mask per bar; grid linesand labels get weak masks and drop out

Before any language model is involved, classical vision finds the candidates. We binarize the image with Otsu thresholding on both RGB and HSV, invert if the background is dark, extract contours, and split each contour by unique pixel value so a group of touching bars separates into three. Solidity and area thresholds drop stray shapes. Because these heuristics are brittle on low-contrast charts and can pick up grid lines or labels, we sample \(n\) points inside each candidate and hand them to the Segment Anything Model (SAM), which returns a tight mask per bar and only weak masks for grid lines and text, so those fall away. For this chart the result is eighteen masks.

3. Stamp a label on each element

Number of documents required per shipment to import goods 2005 2006 2007 0 2 4 6 8 Documents required 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Algeria Angola Antigua and Barbuda Argentina Armenia Australia Set of marks 18 elements, 18 labels. Label k is drawn on bar k. The model never sees pixel coordinates; it refers to bars by label only. Algeria = 1, 2, 3 | Angola = 4, 5, 6 ... | Armenia = 13, 14, 15 | Australia = 16, 17, 18

Each mask gets an alphanumeric label drawn directly on the image, following set-of-marks (SoM) prompting. Reading left to right, Algeria’s three bars become 1, 2, 3, Angola’s 4, 5, 6, and so on through Australia’s 16, 17, 18. From here on the model never has to name a position in pixels; it names a label, and we already know which mask that label points to.

Algeria 2005Algeria 2006Algeria 2007...Australia 2007→123...18

4. The model checks the answer against the chart

Prompt Marked chart image (18 labels) What attribution means Few-shot text examples of question, answer, attribution Question + "Ans: 3" Think step by step: validate, then attribute MLLM (GPT-4o) Step 1: validation Read the 2005 bars (labels 1, 4, 7, 10, 13, 16): 9, 8, 5, 6, 6, 7 documents Mean over six countries = 41 / 6 = 6.83 Above the mean: 9, 8, 7 (three bars) Answer 3 is consistent with the chart continue to step 2

The marked image goes to the MLLM (GPT-4o in our experiments) with a prompt that explains what chart attribution is, gives a few textual examples of question, answer, and attribution, and asks for chain-of-thought reasoning in two steps. Step one is validation: is the answer consistent with the chart? For our question the model has to read the six 2005 bars, take their mean, and count how many exceed it.

1: 94: 87: 510: 613: 616: 7→mean 6.83→9, 8, 7 above→count 3 ✓

5. The model names the marks that support it

Number of documents required per shipment to import goods 2005 2006 2007 0 2 4 6 8 Documents required 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Algeria Angola Antigua and Barbuda Argentina Armenia Australia Step 2: attribution "Supporting elements: 1, 4, 16" Algeria 2005, Angola 2005, Australia 2005 3 highlighted bars = Ans: 3 A reader checks the answer against three bars, not eighteen.

Step two is attribution: which labeled elements support the answer? The model returns 1, 4, and 16, the 2005 bars for Algeria, Angola, and Australia. We map those labels back to their SAM masks and highlight them. The reader’s verification problem has shrunk from “read this chart” to “count these three bars”, and the count matches the answer.

Ans: 3→MLLM→1416→3 bars, consistent

6. The same pipeline on a wrong answer

Number of documents required per shipment to import goods 2005 2006 2007 0 2 4 6 8 Documents required 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Algeria Angola Antigua and Barbuda Argentina Armenia Australia Question In which year did Armenia have the maximum number of documents required per shipment to import goods? Ans: 2006 Validation: inconsistent Armenia: 6, 6, 8 documents, so the maximum is 2007 (15), not 2006. Attribution: 14. dashed = the bar that would be right

Now the second question from Figure 1: “In which year did Armenia have the maximum number of documents required per shipment to import goods?” with the answer 2006. Armenia’s bars read 6, 6, 8, so the maximum is 2007, and the validation step flags the response as inconsistent. The attribution still grounds the answer to the bar it depends on, Armenia’s 2006 bar (mark 14), and one glance shows it is not the tallest of the three. That is what a chart-response misalignment looks like when it is made visible.

Ans: 2006→MLLM→inconsistent14→15 is taller
The same two cases as drawn in Figure 1 of the paper. (1) Attribution grounds each response to specific bars. (2) Verification: the count of 3 is consistent with the highlighted 2005 bars; the answer 2006 for Armenia is inconsistent with the highlighted bar.

Under the hood

Symbol Meaning
\(c \in \mathcal{C}\) A chart image, \(c = \mathcal{I}^{w \times h \times 3}\)
\(v \in \mathcal{R}_c\) A response associated with chart \(c\) (a question and its answer)
\(\mathcal{A}_{c,v}\) The attribution set: chart regions that support \(v\)
\(a_i\) One region, corresponding to an element of \(c\) (bar, sector, point, line segment)
\(f\) The attribution function ChartLens implements
\(n\) Number of points sampled per candidate as SAM prompts
\(D, G\) Detected regions (after IoU matching) and ground-truth regions

The task is post-hoc fine-grained visual attribution for charts. Given a chart and a response, produce

\[\mathcal{A}_{c,v} = \{a_1, a_2, \ldots, a_n\}\]

where each \(a_i\) is a distinct chart element that supports \(v\), subject to relevance (each \(a_i\) bears on \(v\)), completeness (\(\mathcal{A}_{c,v}\) covers all the evidence needed to justify \(v\)), and precision (no irrelevant parts of the chart). The whole method is a mapping

\[f : (c, v) \mapsto \mathcal{A}_{c,v}\]

and the design question is how to make an MLLM implement \(f\) when it cannot reliably output coordinates. ChartLens answers by turning region selection into label selection: the candidate set of elements is computed by segmentation, and the model chooses a subset of labels.

Chart c + response v (question, answer) Bars, pie sectors Otsu + contours (pies: unroll radially) then SAM point prompts Lines LineFormer, then split into equal x-segments Marked image one label per element (SoM) MLLM few-shot + CoT 1. validate 2. attribute lines: pairs of marked points A labels Scoring: labels map back to masks; bars and sectors match ground truth at IoU >= 0.9 and are scored with P, R, F1
The two stages. Mark generation produces one referable element per bar, sector, or line segment; attribution prompts an MLLM with the marked image and reads back the labels it cites.

Mark generation. Bars follow the pipeline in step 2. Pie charts take the largest contour of the binarized image, fit its minimum enclosing circle, unroll the pie along the radial axis into a strip, and detect complete edges in that strip as sector boundaries, which map back to slices. Both then pass through SAM with sampled point prompts (facebook/sam-vit-large). Lines are thin, overlapping, and intersecting, so contour heuristics do not apply; we use LineFormer, a transformer-based line extractor, and divide each recovered line into equally spaced segments along its horizontal extent so that a point on a line has a mark to refer to.

Attribution prompt. The prompt has three parts: a description of chart attribution, few-shot textual examples of question-answer pairs with their attributions, and an instruction to reason step by step through validation (is the QA pair consistent with the chart?) and then attribution (which labeled elements support the answer?). For line charts the model returns pairs of marked points between which the attributed span lies; those pairs are treated as bounding-box corners.

Scoring. For bars and sectors, a predicted region counts as a match if it overlaps a ground-truth region at \(\text{IoU} \geq 0.9\). With \(D\) the matched detections and \(G\) the ground truth,

\[P = \frac{|D \cap G|}{|D|}, \qquad R = \frac{|D \cap G|}{|G|}, \qquad F1 = \frac{2 \cdot P \cdot R}{P + R}.\]

Lines are scored by detection rate (the fraction of ground-truth points covered, a recall-like measure) and by the percentage of the chart’s area the attribution covers, because a method can reach high detection by painting most of the chart.

ChartVA-Eval. To measure any of this we built a benchmark of 1,244 queries over three subsets. ChartVA-AITQA renders synthetic charts from airline SEC-filing tables (301 queries, 203 bar and 98 line charts, one attribution each). ChartVA-PlotQA uses synthetic scientific charts from World Bank Open Data, Open Government Data, and the Global Terrorism Database (595 queries, 396 bar and 199 line charts, 2.4 attributions on average, up to 12). ChartVA-ChartQA uses real charts from Statista, Pew Research, Our World in Data, and OECD, with pie charts oversampled (348 queries: 121 bar, 109 pie, 118 line). For the last two, GPT-4o drafted attributions from the underlying tables with template-specific prompts, and three annotators verified relevance and completeness with Cohen’s kappa of 0.89 and 0.84.

What the numbers say

Baselines are zero-shot GPT-4o bounding-box prompting, Kosmos-2, and LISA; ChartLens uses GPT-4o as its MLLM.

Method Bar: AITQA F1 Bar: PlotQA F1 Bar: ChartQA F1 Pie: ChartQA F1
Zero-shot GPT-4o 22.77 3.30 7.75 7.17
Kosmos-2 0.51 1.01 3.13 11.70
LISA 1.62 0.34 1.01 2.41
ChartLens 69.28 34.65 64.14 48.56

On line charts ChartLens covers 59.14%, 51.84%, and 77.8% of ground-truth points on the three subsets while flagging 1.25%, 9.98%, and 5.34% of the chart area; LISA and Kosmos-2 reach high detection mainly by covering 27% to 63% of the chart, so ChartLens uses 3 to 50 times less area. Across chart types this is a 26-66% improvement in fine-grained attribution over the baselines. Qualitatively, GPT-4o tries to be specific but cannot localize through text coordinates, and LISA and Kosmos-2 return generic components such as the whole pie regardless of the question.

Try it