Glonce

PolicyGuide compiles policy into workflow graphs for compliant LLM agents

researchThe Glonce editorsHugging Face🔥79Glonce rating79Impact60Novelty72Relevance100Surprise40Players45Credibility85
PolicyGuide compiles policy into workflow graphs for compliant LLM agents

Customer-service LLM agents that act on a user's behalf are expected to follow organizational policy, and compliance failures come from two different directions: forbidden actions, such as granting a change the user is not eligible for, and omitted procedural requirements, such as skipping identification or confirmation. The paper argues that existing safeguards cover only part of this. Runtime safeguards can intervene on a risky action as it happens, but action-local checks do not guide an agent through a multi-step procedure, and workflow-following systems keep an agent on a prescribed process but are built to get the workflow completed rather than to safeguard the agent's behavior along the way.

PolicyGuide is built to close that gap. It compiles each domain's policy into a workflow graph, then invokes a proactive verifier at user-turn boundaries, meaning at every point the user speaks rather than only when the agent is about to take a risky action. Working from the graph state it has persisted, the verifier reconciles any open requests and returns step-specific remediation along a policy-compliant path, telling the agent what to do next to get back in line with policy rather than simply blocking a single move.

The authors test PolicyGuide across the τ²-bench airline, retail, and telecom domains using a GPT-5.4 agent paired with a GPT-5.4 verifier. It raises the mean Pass^4 score from 0.42 to 0.62, with the largest gain on telecom, which the paper describes as the most workflow-structured of the three domains: Pass^4 there rises from 0.19 to 0.61. The same workflows, the authors report, transfer to agents built on Claude Sonnet 4.6 and Gemini 2.5 Pro, though the text does not give Pass^4 figures for those two models.

Beyond the main benchmark, the authors report complementary evaluations in which PolicyGuide produces the lowest attack-success rate observed under adversarial users, and the strongest procedural compliance in a workflow-level validation the authors designed themselves. The text states only that these results are the lowest and the strongest, without giving numeric values for either.

Key facts

  • PolicyGuide compiles each customer-service domain's policy into a workflow graph and runs a proactive verifier at user-turn boundaries, rather than checking only individual risky actions.
  • Tested with a GPT-5.4 agent and verifier across the τ²-bench airline, retail, and telecom domains, PolicyGuide raises the mean Pass^4 score from 0.42 to 0.62.
  • The largest gain is on telecom, the domain the paper calls the most workflow-structured of the three: Pass^4 there rises from 0.19 to 0.61.
  • The same workflows built for the GPT-5.4 agent transfer to agents built on Claude Sonnet 4.6 and Gemini 2.5 Pro.
  • Complementary evaluations report the lowest attack-success rate observed under adversarial users and the strongest procedural compliance in an author-designed workflow-level validation, though the text gives no numeric values for either.

Why it matters

Customer-service LLM agents that act on a user's behalf can fail policy in two different ways: taking a forbidden action, such as granting a change the user is not eligible for, or skipping a required procedural step, such as identification or confirmation. The paper argues the existing tooling only half covers this. Runtime safeguards can stop a risky action in the moment, but checking one action at a time does not guide an agent through a multi-step procedure; workflow-following systems keep an agent moving through a prescribed process, but they are built to get the workflow finished, not to safeguard the agent's behavior along the way. PolicyGuide is positioned to cover the gap between those two: it turns a domain's policy into a workflow graph and checks the agent against it at every user turn, not just when the agent is about to act.

Who it affects

The paper frames its target as customer-service LLM agents that act on a user's behalf, and tests the approach on three modeled domains: airline, retail, and telecom. That points to teams building or deploying LLM agents to handle policy-governed customer interactions in comparable domains, where a forbidden action or a skipped confirmation step has real consequences. The evaluation also runs the same workflows against agents built on three different underlying models, GPT-5.4, Claude Sonnet 4.6, and Gemini 2.5 Pro, so the approach is not presented as tied to a single model provider. The text names no specific company or product using PolicyGuide in production.

How to use it

PolicyGuide is presented as a research method rather than a released product: the text gives no pricing, license, or code repository. The design it describes is a two-part recipe: compile the organization's policy for a given domain into a workflow graph, then run a proactive verifier at every user-turn boundary that reads the persisted graph state, reconciles what the user has asked for so far, and returns a step-specific remediation telling the agent how to get back onto a policy-compliant path. The paper also reports that the same workflows transfer to agents built on different underlying models, Claude Sonnet 4.6 and Gemini 2.5 Pro, beyond the GPT-5.4 setup used in the main evaluation.

How solid is it

The evaluation runs across three separate τ²-bench domains, airline, retail, and telecom, with a GPT-5.4 agent and a matching GPT-5.4 verifier, and the paper reports the same workflows also transfer to agents built on Claude Sonnet 4.6 and Gemini 2.5 Pro, which suggests the result is not an artifact of one model. It is also backed by complementary evaluations beyond the main Pass^4 benchmark, an adversarial-user attack-success-rate test and an author-designed workflow-level validation, both reported as PolicyGuide's best results, though without any numbers attached. Several things are missing from the available text: no author names or institutional affiliation are given, no publication or submission date is stated, the Pass^4 metric itself is never defined beyond being the score PolicyGuide raises, and no sample size or test-case count is given for the τ²-bench evaluations, so the scale of the underlying test set cannot be assessed from this source.

Risks and caveats

The reported gain is measured on a benchmark, not in live deployment, and it is uneven across domains: telecom shows the largest jump, from 0.19 to 0.61, while the text gives no separate before-and-after figures for the airline and retail domains beyond the 0.42-to-0.62 mean across all three. The text also does not say what running a proactive verifier at every user turn costs in computation or latency, which matters for a design meant to check in at every turn rather than only when the agent is about to act. And the attack-success-rate and procedural-compliance results, reported only as the best observed, come with no numbers to weigh how large that margin actually is.