Glonce

JevAgentHarness hands agent tool choices to a 4B model

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact45Novelty60Relevance95Surprise35Players15Credibility75
JevAgentHarness hands agent tool choices to a 4B model

LLM agents now fix software, operate computers and serve customers, but the harnesses that run them call a large LLM at every step. That makes long tasks slow and costly. The authors of this preprint note that most of those calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. The question the paper asks is how a harness can hand these decisions to a decision model with little loss of success.

To measure this, the authors build AgentDecisionBench. It tests decision models from isolated questions up to control of a running agent, using public benchmarks and the authors' own DecisionGym environment. Two findings motivate the rest of the work. The decision model that was most accurate on isolated questions solved only 69.5% of agent tasks. And when tool choices were delegated without a fitted confidence threshold, task success fell by 19.0 points.

Building on that, the authors design JevAgentHarness. It fits a confidence threshold for each choice on the agent's own trajectories and lets the LLM decide whenever the decision model's confidence falls below it.

The second contribution is a model. The authors argue that existing decision models never see the states an agent reaches after its own mistakes. So they train AgentDecision-4B, an open 4B model, on such states, with Jev's answers as labels.

On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev, a gap of 9.5 points. On tasks like the training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The authors say the models, data and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.

Key facts

  • AgentDecisionBench tests decision models from isolated questions up to control of a running agent, on public benchmarks and the authors' DecisionGym environment.
  • The decision model most accurate on isolated questions solved only 69.5% of agent tasks; delegating tool choices without a fitted confidence threshold cut success by 19.0 points.
  • JevAgentHarness fits a confidence threshold for each choice on the agent's own trajectories and lets the LLM decide below it.
  • On tasks unlike its training data, the agent solved 91.5% with AgentDecision-4B versus 82.0% with Jev.
  • On tasks like the training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success.

Why it matters

Agent harnesses call a large LLM at every step, so long tasks are slow and costly. Because most of those calls end in a multiple-choice decision, a cheap decision model could take over a large share of them. The paper's contribution is showing where that goes wrong and how to fix it. Accuracy on isolated questions did not predict agent performance: the best model on such questions solved only 69.5% of agent tasks. Delegating tool choices without a fitted confidence threshold cost 19.0 points of success. A per-choice threshold that sends uncertain decisions back to the LLM is the proposed remedy.

Who it affects

Teams that build or run LLM agents for software fixing, computer operation or customer service, and pay for an LLM call at every step. It also concerns researchers who evaluate decision models, since AgentDecisionBench is meant to test them inside running agents rather than on isolated questions.

How to use it

The authors say the models, data and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness. The release is stated in the future tense. The method as described: run the agent, fit a confidence threshold for each choice on the agent's own trajectories, let the decision model answer above the threshold and the LLM below it. AgentDecision-4B is presented as an open 4B model trained with Jev's answers as labels.

How solid is it

These are the authors' own results, reported in a preprint abstract on Hugging Face Papers, and the page shows no independent replication. The figures are concrete: 69.5%, 19.0 points, 91.5% versus 82.0%, 81.1-89.0% cheaper, up to 2.5 times faster, and no loss of 3 or more points of task success. Two different settings are involved: the 91.5% versus 82.0% comparison is on tasks unlike the training data, while the cost and speed figures are on tasks like the training data. The baseline for the cost and speed savings is not stated in the abstract.

Risks and caveats

The abstract does not name the public benchmarks used, the LLM or LLMs in the harness, or the number of tasks. It does not say what Jev's own accuracy, size or provider is. The abstract does not state which 4B models besides AgentDecision-4B are meant by 'models trained this way' or how many. The cost and speed gains apply to tasks like the training data, and the 'up to 2.5 times' figure is a maximum. The abstract names no authors or institutions. The release is promised, not yet confirmed as available.

“Most of these calls end in a multiple-choice decision, such as which tool to call next.”

— Paper abstract, Hugging Face Papers 2610.21412