Glonce

InflationAgent tops FrugalGPT with 31% fewer tokens on GSM8K

researchThe Glonce editorsarXiv🔥74Glonce rating74Impact70Novelty75Relevance95Surprise60Players25Credibility85

Agentic LLM systems retry a failed query, and each retry burns more tokens. Researchers name this gap between a model's advertised per-token price and what a full multi-step workflow actually costs "token inflation", defined as the ratio of true workflow cost to single-call cost. Existing cost-aware routers such as FrugalGPT pick a model based on single-call cost alone, which the authors show can underestimate real cost by more than 2x on difficult tasks.

To address this, the paper presents InflationAgent, a four-stage router. First, it measures token inflation across model tiers and task types, finding inflation as high as 4.25x for a 7B-parameter model on multi-hop question answering. Second, it introduces CoT Branching Entropy (CBE), a difficulty signal computed before execution entirely from local inference, which predicts high inflation with an AUROC of 0.887. Third, it selects which model to route a query to by maximizing a Semantic Exchange Rate (SER), a ratio of expected accuracy to predicted true cost, rather than raw per-token price. Fourth, it applies a fresh-escalation policy: when a chain fails, the system discards it and starts over with a stronger model instead of forwarding the failed reasoning as context.

On GSM8K under a fixed token budget, InflationAgent reaches 94.7% accuracy versus 91.0% for FrugalGPT, while using 31% fewer tokens. The authors also test what happens without fresh-escalation: forwarding a failed reasoning chain to GPT-4o instead of letting it start fresh cuts GPT-4o's accuracy by up to 34.8 percentage points, which they present as the evidence that discarding failed chains rather than forwarding them is the right design choice.

Key facts

  • Token inflation is defined as the ratio of true agentic-workflow cost to single-call cost; routers like FrugalGPT that ignore it can underestimate real cost by more than 2x on hard tasks.
  • Measured token inflation reaches as high as 4.25x for a 7B-parameter model on multi-hop question answering.
  • CoT Branching Entropy (CBE), a difficulty signal computed from local inference before execution, predicts high inflation with an AUROC of 0.887.
  • On GSM8K under a fixed budget, InflationAgent hits 94.7% accuracy versus FrugalGPT's 91.0%, using 31% fewer tokens.
  • Forwarding a failed reasoning chain to GPT-4o instead of starting fresh cuts its accuracy by up to 34.8 percentage points, the paper's justification for discarding failed chains before escalating to a stronger model.

Why it matters

Per-token pricing is the number teams budget against, but an agentic system that retries after failures spends far more than one call's worth of tokens. The paper shows that ignoring this gap, as single-call-cost routers like FrugalGPT do, can understate the real bill by more than 2x on the hardest tasks, and by as much as 4.25x for a small 7B model on multi-hop question answering. That is the practical case for routing decisions to account for expected retries, not just the price of the first attempt.

Who it affects

Anyone building or paying for agentic LLM pipelines that route between models of different sizes and prices, and anyone using single-call token price as a proxy for what a multi-step workflow will actually cost. The comparison baseline, FrugalGPT, is itself a cost-aware routing system, so the result speaks directly to teams already trying to optimize spend across model tiers.

How to use it

InflationAgent is presented as a research system, not a shipped product or service; the source gives no pricing, license or availability details. The reusable ideas are the CoT Branching Entropy signal, which flags a likely-difficult query before it runs using only local inference, and the Semantic Exchange Rate, which ranks candidate models by expected accuracy divided by predicted true cost rather than by list price. The fresh-escalation policy, discarding a failed chain instead of feeding it to a stronger model, is the other concrete design choice the results support.

How solid is it

The headline comparison against FrugalGPT is on one benchmark, GSM8K, under a fixed token budget, where InflationAgent's 94.7% beats FrugalGPT's 91.0% while using 31% fewer tokens. The inflation measurements and the CBE signal's 0.887 AUROC are evaluated across multiple model tiers and task types, including multi-hop question answering. The source text names no authors or institutions and gives no publication date, so provenance cannot be checked from the material available here.

Risks and caveats

The source states no dollar-cost figures anywhere, only ratios, percentages and percentage points, so absolute savings cannot be estimated from this text. The 7B model used in the inflation measurements is identified only by parameter count, not by name. GPT-4o's baseline accuracy before the 34.8-point drop is not given, only the size of the drop. The text describes results on GSM8K and multi-hop question answering specifically; no broader set of benchmarks is mentioned.