Abstract

Current AI-powered code assistance tools often struggle with poorly-defined problem statements that lack sufficient task context and requirements specification. Recent analysis of software engineering agents reveals that failures on such underspecified requests are highly correlated with longer trajectories involving either over-exploration or repeated attempts at applying the same fix without proper evolution or testing, leading to suboptimal outcomes across software development tasks. We introduce CodeScout, a contextual query refinement approach that systematically converts underspecified user requests into comprehensive, actionable problem statements through lightweight pre-exploration of the target codebase. Our key innovation is demonstrating that structured analysis before task execution can supplement existing agentic capabilities without requiring any modifications to their underlying scaffolds. CodeScout performs targeted context scoping, conducts multi-perspective analysis examining potential fixes and exploration opportunities, then synthesizes these insights into enhanced problem statements with reproduction steps, expected behaviors, and targeted exploration hints. This pre-exploration directly addresses the identified failure patterns by reducing non-converging agent trajectories while clarifying user intent in natural language space. We evaluate CodeScout using state-of-the-art agentic scaffolds and language models on SWE-bench Verified, demonstrating a 20% improvement in resolution rates with up to 27 additional issues resolved compared to the default baseline method. Our results suggest that systematic query refinement through contextual analysis represents a promising direction for enhancing AI code assistance capabilities.

The problem

Developers write short, context-dependent issue reports. They skip reproduction steps, technical details, and clear expectations because they assume the reader already knows the codebase. LLM-based software agents are the opposite: they work best with explicit, well-scoped specifications. Prior studies found that resolvable bug reports have far higher description quality than unresolvable ones, and that agent failures follow two recognizable patterns: over-exploration, where the agent never reaches the root cause, and repeatedly applying the same fix without testing or evolving it.

We argue that agents should look before they leap. Instead of asking the agent to discover the codebase incrementally inside its reason-execute-observe loop, we invest a small amount of computation up front to build a comprehensive picture of the problem. Figure 1 in the paper shows the effect on a Django issue: the original problem statement leads to 21 steps of unfocused exploration and failure, while the enhanced statement resolves the issue in 6 steps.

Approach

CodeScout is a plug-and-play preprocessing step. It takes the original problem statement and the repository and produces an augmented specification, with no changes to the downstream agent.

  • Repository Knowledge Graph Construction. An AST visitor parses the repository into a directed graph whose vertices are code entities (classes, functions, imports, variables) and whose edges capture inheritance, import and module relationships.
  • High Level Scoping. Given the problem statement and the repository graph, an LLM agent proposes 5 to 10 exploration targets (files, classes, functions), each with a reason for its relevance. Retrieval happens directly from the graph, so this stage does not need full source access.
  • Fine-grained Context Analysis. For each target, the source is retrieved and analyzed for its role in the issue, fix location hints with confidence estimates, technical insights, and alternative hypotheses. The LLM also assigns a relevance score, and only targets above a threshold survive.
  • Problem Synthesis. An LLM combines the original statement with the filtered insights into a structured document: an enhanced issue description, reproduction steps, expected behavior, exploration hints, and fix hints.

In practice each problem statement costs roughly 9 LLM calls: one for scoping, one per surviving target (about 7 valid targets per instance), and one for synthesis.

Original issue short, no repro, no fix location 1. Scoping LLM picks 5-10 files / classes / functions from the repo graph Repository knowledge graph 2. Analysis per target: role, fix hints, hypotheses, relevance score filter: score > threshold auth/forms.py kept target 2 kept target n dropped 3. Synthesis enhanced issue + repro steps + expected behavior + exploration/fix hints Agent unchanged ~9 LLM calls
CodeScout runs before the agent: scope the repository graph, analyze and score each target, synthesize an enhanced problem statement, then hand it to an unmodified software agent.

Example

The paper’s Figure 1 walks through SWE-bench Verified instance django__django-11790 with GPT-5-mini in mini-SWE-agent. The original issue states the symptom but gives the agent nothing to start from; the augmented version adds a reproduction test, the bug location, and a fix snippet. The text below is transcribed from the figure (trimmed with “…”).

Input: original problem statement

# AuthenticationForm's username field doesn't set maxlength HTML attribute.

# Description
AuthenticationForm's username field doesn't render with maxlength HTML
attribute anymore. Regression introduced in #27515 and
5ceaf14686ce626404afb6a5fbd3d8286410bf13.
https://groups.google.com/forum/?...#!topic/django-developers/qnfSqroODlA
https://forum.djangoproject.com/t/possible-authenticationform-max-length-regression-in-django-2-1/241

Agent trajectory: 21 steps of grep, nl, and repeated patch attempts. Not resolved. No reproduction steps, no bug origin, unclear expectations.

Output: CodeScout-augmented problem statement (excerpt)

# AuthenticationForm Username Field maxlength HTML Attribute Regression

## Issue Description
The `AuthenticationForm`'s username field no longer renders with the HTML
`maxlength` attribute. This regression was introduced in PR #27515 (commit
5ceaf14686ce626404afb6a5fbd3d8286410bf13). While the form correctly sets the
`max_length` Python property on the username field, this value is not being
propagated to the HTML `maxlength` attribute during rendering. This creates a
disconnect between server-side validation (which still enforces the maximum
length) and client-side validation/constraint (which no longer limits input
length in the browser).

## Reproduction Steps
1. Create a Django authentication form in a view:
   from django.contrib.auth.forms import AuthenticationForm

   def login_view(request):
       form = AuthenticationForm()
       return render(request, 'login.html', {'form': form})
   ...

## Expected Behavior ...
## Exploration Hints
### Files to Examine: ...
### Key Classes/Functions: ...

## Fix Hints
### High Confidence Locations:
- `AuthenticationForm.__init__()` in `django/contrib/auth/forms.py` - The most
  direct fix would be to add explicit setting of the HTML attribute where the
  Python property is already being set

### Implementation Hints:
- Add the following code in the `__init__` method of `AuthenticationForm`
  after setting `max_length`:
  self.fields['username'].widget.attrs['maxlength'] = self.username_field.max_length or 254

Agent trajectory: 6 steps. The agent opens forms.py, applies the patch, runs the reproduction snippet, and submits. Resolved.

The same instance as rendered in the paper's Figure 1. Left: the original problem statement leads to 21 steps and no resolution. Right: the CodeScout-augmented statement includes a reproduction test, the bug location, and a fix snippet, and the agent resolves the issue in 6 steps.

Results

We evaluate on SWE-bench Verified with three scaffolds (SWE-agent, OpenHands, mini-SWE-agent) and three models (GPT-5-mini, DeepSeek R1, Qwen3 Coder 480B). Augmentation improves resolution rates across all scaffold and model combinations, with the largest gains when the runtime agent is weaker; overall this is a 20% improvement in resolution rate with up to 27 additional issues resolved. The ablation below uses SWE-agent as the scaffold.

Method (SWE-agent scaffold) DeepSeek R1 GPT-5-mini Qwen3 Coder
Default (no augmentation) 114 194 183
CodeScout 125 209 207
Agentic intra-trajectory augmentation 109 177 158
CodeScout without relevance filtering 116 190 190
CodeScout with BM25 entity selection 119 195 198

Source: Table 1 of the paper. Numbers are counts of resolved issues on SWE-bench Verified with SWE-agent. Higher is better.

  • CodeScout adds +11 (+9.6%), +15 (+7.7%), and +24 (+13.1%) resolved issues for DeepSeek R1, GPT-5-mini, and Qwen3 Coder respectively.
  • Asking the agent to augment the problem statement itself during its trajectory hurts: resolved counts drop below the default baseline by 5, 17, and 25 issues. A separate, structured pre-exploration stage is what delivers the gain.
  • Relevance filtering is necessary; without it most of the benefit disappears. Replacing LLM scoping with BM25 retrieval still helps but yields smaller gains.
  • File- and function-level localization also improves across models, most notably for DeepSeek R1.

Cross-synthesis. The model that writes the problem statement need not be the model that runs the agent.

Runtime model Default Augmented by DeepSeek R1 Augmented by Qwen3 Coder Augmented by GPT-5-mini
DeepSeek R1 108 125 164 132
Qwen3 Coder 183 194 209 190
GPT-5-mini 194 196 207 209

Source: Table 2 of the paper. Counts of resolved issues on SWE-bench Verified; bold marks the best augmenter for each runtime model. Higher is better.

  • A stronger model can write problem statements for a weaker agent: DeepSeek R1 as the runtime model goes from 108 to 164 resolved issues (+56, +51.9%) when Qwen3 Coder does the augmentation.
  • A stronger runtime model gains only modestly from a weaker augmenter (GPT-5-mini: 194 to 196, +1.0%), so augmentations can be pre-computed once with a capable model and reused.
  • For the same token budget, augmented runs resolve more issues for Qwen3 Coder and DeepSeek R1, even after charging the augmentation overhead to the agent. GPT-5-mini’s trajectories are an order of magnitude longer, so the overhead is negligible but its absolute token budget remains high.
  • Trajectory analysis shows agents start with more view and grep calls and fewer find calls under augmentation, consistent with more targeted early exploration.

Resources