Glonce

HALO benchmark finds AI agents act too early and too much

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact40Novelty55Relevance95Surprise30Players10Credibility75
HALO benchmark finds AI agents act too early and too much

A preprint introduces HALO, short for Human-Grounded GrAnular EvaLuation of PrOactive Agents. It targets a specific problem: proactive agents must decide whether, when, and how far to assist without an explicit request. The authors say existing evaluations often rely on automated scores or binary verdicts, which offer limited human diagnosis of evidence use and intervention quality.

The benchmark, HALO-Arena, covers 4 proactive need types: Dependency, Blockage, Risk, and Opportunity. Scenarios span 10 everyday domains and 50 personal environments. In each one, the agent has to retrieve evidence from structured databases or unstructured documents before it judges whether assistance is warranted and chooses how far to go.

On top of the benchmark, the authors build HALO-8K, a dataset of 8,000 responses from 10 models across 400 scenarios. Twenty experts annotated each response for action level, an acceptability verdict, and human diagnosis. For rejected responses, the diagnosis covers timing, intensity, and other issues. The authors also recorded reasoning tokens and retrieval rounds to measure the effort behind the responses.

The results are modest. The best proprietary model reaches 72.88% acceptability on HALO-Arena, and the best open-source model reaches 56.63%. The authors' central finding is that over-proactivity is the dominant observed failure mode: Premature timing or Excessive intensity accounts for 61.20% of rejected responses. They add that further analyses yield complementary findings on the balance between intervention and restraint, evidence acquisition, and the calibration of timing and intensity.

Key facts

  • HALO-Arena tests proactive agents on 4 need types (Dependency, Blockage, Risk, Opportunity) across 10 everyday domains and 50 personal environments.
  • HALO-8K holds 8,000 responses from 10 models over 400 scenarios, annotated by 20 experts for action level, acceptability verdicts, and human diagnosis.
  • The best proprietary model scores 72.88% acceptability; the best open-source model scores 56.63%.
  • The central finding is that over-proactivity is the dominant observed failure mode: Premature timing or Excessive intensity accounts for 61.20% of rejected responses.

Why it matters

Agents that act without being asked have to get three things right: whether to step in, when, and how far. The authors argue that current evaluations lean on automated scores or binary verdicts, which say little about why an intervention was good or bad. HALO adds expert human diagnosis of timing and intensity. Its headline result is that when proactive agents fail, they most often do too much or do it too soon, rather than staying silent.

Who it affects

Teams building assistants that offer help unprompted are the obvious audience, along with researchers who evaluate such agents. The scenarios sit in everyday domains and personal environments, so the findings bear on consumer-facing assistants more than on narrow enterprise tools.

How to use it

The source describes HALO-Arena as a benchmark and HALO-8K as an annotated dataset of responses, so they can serve as a test bed for judging proactive behaviour. Scenarios require evidence retrieval from structured databases or unstructured documents, so an agent under test needs retrieval ability. The timing and intensity labels on rejected responses point to where to tune an agent.

How solid is it

The setup has real scale for an annotation study: 8,000 responses, 400 scenarios, 10 models and 20 expert annotators. The reported percentages are best results per group, not averages. The source is an abstract-level summary: it does not name the 10 models or say which ones led, and it does not split the 61.20% between premature timing and excessive intensity. It gives no figures for reasoning tokens or retrieval rounds.

Risks and caveats

The over-proactivity finding is described as the dominant observed failure mode, which ties it to the models and scenarios tested here. Acceptability rests on expert verdicts, and the text does not say how it is defined numerically beyond that. With the evaluated models unnamed, the 72.88% and 56.63% figures cannot be matched to specific products from this text.