Ask fewer, better questions: SAGE-Agent and ClarifyBench explained
You hand your phone to an assistant and say “Hey! can you call Alex from work for me?” Your contacts list has two people named Alex: Alex C., who has a work number, and Alex M., who has a home number. A careful human glances at the list and dials Alex C. A typical LLM agent does one of two things instead. It asks “Which Alex do you want to call?”, a question the contacts list already answers, or it dials whoever comes first.
Both failures have the same root. The agent reasons about ambiguity in the space of words: it is prompted to write a clarifying question as free text, so it has no explicit picture of which argument is missing or whether the tool’s own schema already pins it down. In our ACL 2026 Findings paper, Structured Uncertainty Guided Clarification for LLM Agents, we with collaborators at Adobe Research and UMD move the uncertainty into the space of tool parameters and their domains. Once it lives there, “should I ask?” and “what should I ask?” become quantities the agent can compute.
The idea in one picture
Walkthrough: calling Alex, then texting Maya
The walkthrough traces both requests through SAGE-Agent (Structured Argument Uncertainty guided Elicitation Agent). The belief values are worked by hand from the paper’s Equation 2 with the settings we used in the experiments, a redundancy penalty of \(\lambda = 0.5\) and a stopping coefficient of \(\alpha = 0.1\).
1. The request, the contacts, and the tool schema
Everything the agent needs is already on the table. The request (blue) names a person and a label. The contacts list is the domain of the contact argument: a finite set of four phone entries. The tool schema (grey) says which arguments each tool takes and which are required. SAGE-Agent never reasons about ambiguity without this schema in view.
2. Propose candidate tool calls with unknowns
An LLM reads the request and the schema and proposes candidate tool calls, filling each argument with a value or <UNK>. What it says in the query becomes constraints on the argument’s domain (the paper’s Update operation), and the contact domain shrinks:
The word “call” also settles the tool. The paper uses a uniform prior over tools, so nothing forces Call over SMS a priori; it is the candidate generator that proposes only Call here.
3. Turn the domains into a belief over calls
Here is the move that makes the rest computable. A candidate with an unspecified argument is spread evenly over that argument’s remaining domain, so its per-argument certainty is one over the domain size. For Alex, the domain has one entry, so p(contact) = 1 and the belief in Call([Alex C. (Work)]) is 1.0. Because the belief is defined over the schema and not over generated text, it measures what the user has left unsaid (specification uncertainty) separately from what the LLM might misread (model uncertainty).
4. Score the candidate question by EVPI minus cost, then decide not to ask
The LLM also drafts candidate questions, each tagged with the candidate it targets and the aspects (tool, parameter pairs) it would resolve. For each one we simulate a perfect answer and ask how much the probability of the best candidate would rise in expectation. That is the question’s EVPI. “Which Alex do you want to call?” targets (Call, contact), whose domain is already a single entry, so the answer cannot move the belief and EVPI is exactly zero. The score falls below the stopping bar, so SAGE-Agent executes:
5. "Text Maya": a domain of size two, so the agent asks
The follow-up request lands on the same machinery. “Maya” narrows the contact domain to two entries, and nothing in the request chooses between them, so the belief splits evenly:
Whichever way the user answers “Should I text her on her home phone?”, the domain collapses to one entry and the best candidate’s probability jumps from 0.5 to 1.0. That gives the question an EVPI of 0.5, far above the stopping bar, so this time SAGE-Agent asks. The token-space baseline, which never sees that the domain has two entries, sends the text to Maya’s default number.
6. Fold the answer into the domain, then execute
The answer is not pasted into a prompt and forgotten. It becomes a constraint on the contact domain, the belief is recomputed, and the counter for the aspect (SMS, contact) goes up by one. Belief 1.0 clears the execution threshold, and the agent sends the text to the right number:
If the tool call had failed at runtime, the agent would propose a corrected call or an error-specific question and re-enter the scoring step (step 4) rather than retrying blindly.
Under the hood
The clarification loop is a sequential decision problem: at each turn the agent chooses between executing its best candidate and asking a question, with the user’s true intent hidden. The paper frames this as a POMDP whose observations are user answers and whose belief state is a distribution over structured tool calls, and uses a Bayesian value-of-information objective to choose questions.
| Symbol | Meaning |
|---|---|
| \(T_i = (name_i, \Theta_i, \mathcal{D}_i, \mathcal{R}_i)\) | Tool schema: parameter set, per-parameter domains, required parameters |
| \(c\) | Candidate call: a partial assignment of a tool’s parameters; unspecified ones are <UNK> |
| \(\mathcal{C}_t, \mathcal{Q}_t\) | Candidate calls and candidate questions proposed by the LLM at step \(t\) |
| \(\pi_c(t)\) | Belief that candidate \(c\) matches the user’s intent after answers \(r_{1:t}\) |
| \(\mathcal{D}_{c,j}(t)\) | Remaining domain of parameter \(j\) of candidate \(c\) |
| \(a = (T_i, \theta_j)\) | An aspect: one parameter of one tool; \(\mathcal{A}(q)\) is the set a question targets |
| \(n_a(t)\) | How many times aspect \(a\) has been asked about |
| \(\lambda, \alpha, \tau_{\mathrm{exec}}, \epsilon\) | Redundancy penalty, stopping coefficient, execution threshold, certainty for continuous domains |
Belief. With a uniform prior over tools and conditional independence across parameters, the belief in a candidate factors over its parameters:
\[\pi_c(t) \propto \prod_{j=1}^{m_c} p(\theta_{c,j} \mid T_c, u, r_{1:t}), \qquad p(\theta_{c,j}) = \begin{cases} 1 & \text{specified} \\ \lvert \mathcal{D}_{c,j}(t) \rvert^{-1} & \text{unspecified, finite domain} \\ \epsilon & \text{unspecified, continuous domain} \end{cases}\]An answer \(r_t\) to question \(q_t\) updates each targeted domain by intersection, \(\mathcal{D}_\theta(t+1) = \mathcal{D}_\theta(t) \cap \mathrm{Update}(\theta, r_t, q_t)\), and the beliefs are renormalized. Answers can be explicit values, cross-parameter dependencies, or exclusions such as “not business class”; all three are domain constraints, which is what keeps the update exact.
Value of a question. The Expected Value of Perfect Information of \(q\) is the expected gain in best-candidate certainty if the aspects it targets were resolved perfectly:
\[\mathrm{EVPI}(q, \mathcal{B}(t)) = \mathbb{E}_{r}\!\left[\max_c \pi_c(t \mid q, r)\right] - \max_c \pi_c(t).\]In practice we simulate the resolution directly: for each candidate, multiply \(\pi_c(t)\) by \(\lvert \mathcal{D}_a \rvert\) for every targeted aspect it leaves unspecified, then take the expected maximum. EVPI is non-negative, has diminishing returns over question sequences, and goes to zero as the belief concentrates, which is exactly what happened to “Which Alex?” in step 4.
Cost, selection, stopping. Asking about the same aspect twice is discouraged by a redundancy cost, and the agent picks the question with the best net score:
\[q^*(t) = \arg\max_{q \in \mathcal{Q}_t}\Big[\mathrm{EVPI}(q, \mathcal{B}(t)) - \lambda \sum_{a \in \mathcal{A}(q)} n_a(t)\Big], \qquad \text{execute } c^*(t) \text{ if } \max_q \mathrm{Score}(q,t) < \alpha \cdot \max_c \pi_c(t).\]The agent also executes immediately when \(\max_c \pi_c(t) \geq \tau_{\mathrm{exec}}\), and stops after \(n_s\) steps. Because the stopping bar scales with the current best belief, the loop ends either because the belief has become confident or because no remaining question is worth its cost. In the experiments we set \(\lambda = 0.5\), \(\alpha = 0.1\), and \(\epsilon = 10^{-4}\).
The same belief as a training signal. The certainty \(\max_c \pi_c(t)\) also makes a reward. In GRPO training on 9K When2Call examples, we multiply the action-classification reward by \(\mathrm{Cert}(a_t)\), which is \(\max_c \pi_c(t)\) for a tool call, \(1 - \max_c \pi_c(t)\) for a clarifying question, and 1 otherwise, so confident correct calls get full payoff, low-certainty calls are penalized, and asking is rewarded only when uncertainty is high, with no critic to judge question quality.
Where the evaluation comes from. We built ClarifyBench to test this in multi-turn use: 716 tasks across document processing, vehicle control, stock trading, travel, and file systems, over 92 tools, with an LLM user simulator that holds the true intent, answers questions, and issues follow-up requests. Queries are split into explicit, ambiguous, and infeasible (which should be declined), and every generated query was checked by two annotators.
What the numbers say
| Method (GPT-4o, ambiguous split) | Coverage | TMR | PMR | Avg. #Q |
|---|---|---|---|---|
ReAct + ask_question() | 42.88 | 70.41 | 62.55 | 2.68 |
| ProCOT | 54.27 | 75.62 | 66.82 | 2.07 |
| Active Task Disambiguation | 45.60 | 77.10 | 60.78 | 3.42 |
| Domain-aware ReAct | 55.70 | 79.83 | 68.04 | 2.56 |
| SAGE-Agent, heuristic-based | 56.42 | 82.31 | 69.81 | 1.82 |
| SAGE-Agent | 59.73 | 86.02 | 71.79 | 1.39 |
Coverage counts tool calls that fully match the ground truth; TMR and PMR are tool and parameter match rates; #Q is the mean number of clarifying questions per task. Across baselines and both base models (GPT-4o and Qwen2.5-14B-Instruct), SAGE-Agent reaches 7-39% higher coverage on ambiguous tasks while asking 1.5-2.7x fewer questions. The heuristic row triggers questions on <UNK> alone without EVPI scoring, and gives up 1-3 points across metrics while asking 0.2-0.4 more questions, which is the value of steps 4 and 5 above. Raising \(\lambda\) from 0 to 0.5 cuts questions by 18.1% (ambiguous), 26.6% (explicit), and 24.2% (infeasible) with Coverage, TMR, and PMR within 3%, so the pruned questions were redundant. On the training side, certainty-weighted GRPO lifts When2Call accuracy from 36.5% to 65.2% for Qwen2.5-3B and from 36.7% to 62.9% for Qwen2.5-7B; the 3B model with our reward beats the 7B model with the standard reward (45.1%).
Try it
- Paper page: /papers/sage-clarification/
- arXiv: 2511.08798
- Code and benchmark: github.com/MananSuri27/ClarifyBench
- ACL Anthology: 2026.findings-acl.2028