Glonce

CAPA benchmark tests whether coding AI remembers a user's ambiguity

researchThe Glonce editorsHugging Face🔥71Glonce rating71Impact70Novelty75Relevance95Surprise40Players30Credibility80
CAPA benchmark tests whether coding AI remembers a user's ambiguity

AI coding assistants translate informal requests into working code, but many requests are ambiguous, and for a given user the same kind of ambiguity tends to recur across different sessions. Existing disambiguation approaches handle each ambiguous request in isolation within the current session, typically by asking the user a follow-up question. The researchers behind this work note that it remained an open question whether a user's already-resolved past sessions could serve as memory, letting an assistant handle a new, similarly ambiguous request in a fresh session while asking fewer clarifying questions.

They formalize this as a new task, personalized ambiguity adaptation: given a user's previously resolved coding sessions plus a new ambiguous request, an assistant should identify the recurring ambiguity pattern, produce the intended executable solution, and minimize clarification.

To benchmark the task they built CAPA. It characterizes personalized coding ambiguity through six mechanisms, which a controlled three-stage generation pipeline injects into otherwise unambiguous, executable coding tasks. The result is 600 coding sessions spread across 60 balanced user-ambiguity cells, with 300 of those sessions held out for evaluation.

Using CAPA, the authors evaluate 12 recent LLMs under two conditions: no history and same-user history, scoring them on executable success, first-turn success, and turns-to-completion. Their analysis covers task difficulty, user identity, and how models use memory-based history, and they additionally propose same-user history gating as a lightweight inference-time method.

The authors frame CAPA as a foundation for building longer-term coding assistants that align generated code more closely with what a user actually intends while cutting down on repeated clarification.

Key facts

  • CAPA contains 600 coding sessions across 60 balanced user-ambiguity cells, with 300 held out for evaluation.
  • Personalized coding ambiguity is characterized through six mechanisms, injected into unambiguous executable tasks via a controlled three-stage generation pipeline.
  • 12 recent LLMs are evaluated under no-history and same-user-history conditions, scored on executable success, first-turn success, and turns-to-completion.
  • The task, personalized ambiguity adaptation, asks an assistant to use a user's previously resolved sessions to spot recurring ambiguity and minimize new clarification requests.
  • The authors propose same-user history gating as a lightweight inference-time method alongside the benchmark.

Why it matters

A coding assistant that has already resolved a user's ambiguous request once still asks the same kind of clarifying question again the next time that user phrases a request the same way, because most disambiguation methods treat every session as a blank slate. This paper argues that a user's own session history is a usable signal for resolving that user's recurring ambiguity, and turns the idea into a defined, benchmarkable task rather than a general product aspiration.

Who it affects

The direct audience is researchers and builders of AI coding assistants who need a standard way to measure whether their models actually get better at a specific user over repeated sessions. Indirectly it targets a real friction point for developers who use these assistants daily and currently have to re-explain the same preferences each time a similar ambiguity comes up.

How to use it

CAPA gives model builders a controlled test bed: run an assistant under the no-history condition versus the same-user-history condition on the same 300 held-out sessions and compare executable success, first-turn success, and turns-to-completion. The authors also offer same-user history gating as a lightweight method that can be applied at inference time, on top of an existing model, rather than requiring retraining.

How solid is it

The benchmark's construction is systematic: six named ambiguity mechanisms, a controlled three-stage generation pipeline, and a fixed split of 600 sessions into 60 balanced cells with 300 held out for evaluation, tested against 12 recent LLMs. The available text is the paper's abstract, though, and does not report the actual per-model scores or the size of any improvement from using same-user history, so how well the approach works in practice cannot be assessed from this material alone.

Risks and caveats

The source text does not disclose concrete accuracy or success figures, does not describe what the six ambiguity mechanisms are or how same-user history gating works mechanically, and does not name the authors' affiliations, a publication venue, or a release or availability timeline for CAPA.