Domain terms and action directives cut LLM output variance 40.7% in code generation

Large language models can produce very different outputs when a prompt's wording changes only slightly, even though its meaning stays the same, a problem prior prompt-engineering work has mostly tackled with black-box optimization or coarse templates. This paper instead runs what the authors describe as the first large-scale, n-gram token-level mechanistic analysis of prompt stability, built on a dataset of 132,000 prompt variants, to explain the mechanism behind that sensitivity rather than just work around it.

The analysis surfaces a pattern the authors call the 'Scaling Law of Prompt Performance Stability': prompts that already achieve higher average task performance also tend to show lower variance and greater robustness when their wording is perturbed, meaning accuracy and consistency rise together rather than trading off. Investigating why, the authors identify two linguistic drivers behind that robustness. The first is domain-specific terminology, precise field vocabulary that tightly anchors a prompt's semantic boundaries. The second is explicit action directives, instructions that formalize the reasoning steps a model is meant to follow. Together, the two elements narrow the model's interpretative space, which the authors describe as effectively 'locking in' more deterministic generation behavior, so small wording changes have less room to steer the output somewhere else.

Building on that mechanistic account, the authors introduce an automated Prompt-Refining Agent that rewrites a user's input by injecting domain anchoring and operational constraints, adding the two stabilizing elements to a prompt automatically. In an empirical evaluation on a code generation task, prompts processed by the agent cut performance variance by 40.7% relative to the unrefined baseline, while mean performance was preserved or improved, so the gain in consistency did not come at the cost of accuracy.

Key facts

A mechanistic analysis built on 132,000 prompt variants finds that higher-performing prompts also show lower variance, a pattern the authors call a 'Scaling Law of Prompt Performance Stability.'
Two linguistic features explain much of that stability: domain-specific terminology, which anchors a prompt's meaning, and explicit action directives, which spell out the reasoning steps a model should follow.
An automated Prompt-Refining Agent injects domain anchoring and operational constraints into a prompt to add those two stabilizing features automatically.
On a code generation task, prompts refined this way cut performance variance by 40.7% relative to the unrefined baseline, while mean performance was preserved or improved.
The paper names no authors, institutions, model, benchmark or programming language, and gives no absolute variance figures behind the 40.7% relative reduction.

Why it matters

Prompt brittleness, where rewording a prompt without changing its meaning shifts a large language model's output, is a familiar practical headache that prior work has mostly attacked with black-box optimization or coarse templates. This paper instead runs what the authors call the first large-scale, n-gram token-level mechanistic analysis of prompt stability, built on a dataset of 132,000 prompt variants, to explain the mechanism behind that sensitivity rather than just work around it. The headline finding, a 'Scaling Law of Prompt Performance Stability,' is that higher average task performance and lower output variance move together rather than trading off, and the authors trace that stability to two specific linguistic features rather than treating it as an unexplained property of a prompt. That mechanistic account is what lets the paper move from observing brittleness to fixing it: an automated agent built on the two features cuts output variance by 40.7% on a code generation task without sacrificing mean performance.

Who it affects

Anyone writing or generating prompts for production LLM systems, including developers building code-generation tools, agent pipelines and other applications where inconsistent output from equivalent-meaning prompts is costly. It also matters to researchers and toolmakers working on prompt optimization, since the paper offers a mechanistic explanation, precise domain terminology plus explicit action directives, rather than a black-box search procedure.

How to use it

The practical implication sits at the prompt-writing level: prompts that use precise, domain-specific terminology and spell out explicit action directives, formalizing the steps the model should follow, are more robust to small wording changes than vaguer, template-style prompts. The paper's own demonstration of this is the Prompt-Refining Agent, which automatically rewrites an input prompt by injecting domain anchoring and operational constraints. Applied to a code generation task, it cut performance variance by 40.7% while preserving or improving mean performance.

How solid is it

The stability analysis is grounded in a large empirical base, 132,000 prompt variants, which the authors describe as a first-of-its-kind mechanistic study at this scale and granularity. The Prompt-Refining Agent's payoff, a 40.7% cut in performance variance with mean performance preserved or improved, is likewise an empirical result, not a projection. But the source text gives no benchmark name, programming language or model identity for the code-generation evaluation, and no absolute variance figures behind the 40.7% relative number, so it is not possible from the abstract alone to judge how large the underlying swings were.

Risks and caveats

The demonstrated 40.7% variance reduction is reported for a single task, code generation, evaluated with a model and benchmark the text does not name. The mechanism behind the Prompt-Refining Agent's own decisions, which domain terms or action directives it injects into a given prompt and how, is not described beyond the general statement that it 'systematically restructures' input queries by injecting domain anchoring and operational constraints. The paper also does not name its authors, institutions or publication date, which limits any assessment of provenance or peer review status at this stage.