Glonce

Ventor-QTest flags quality loss in vendor-hosted LLM APIs

researchThe Glonce editorsHugging Face🔥73Glonce rating73Impact65Novelty72Relevance100Surprise42Players68Credibility87
Ventor-QTest flags quality loss in vendor-hosted LLM APIs

Researchers propose Ventor-QTest, a composite black-box method for auditing the quality of vendor-hosted inference APIs that serve open-weight large language models. Third-party providers that host open-weight models have become a significant part of the LLM ecosystem, but verifying that an API actually serves the model it claims to, at the quality it claims, has been an open problem, especially since customers of a hosted API typically cannot see the token probabilities a provider would need to share for a rigorous check. The authors formalize hosted model routing as a stochastic process and build a method that needs no probability information from the target API at all.

Ventor-QTest combines two components. The repeated-request component sends the same frozen, constrained prompt to the target API many times, reconstructs a categorical distribution over the returned text from the counts of each response, and reports the result as average fidelity loss (AFL), a null-bias-corrected, within-window mean coarsened-KL statistic. The long-sequence component instead runs many independent long sequences and reports extreme fidelity loss (EFL) as the empirical upper tail of a run-level reference-centered-surprisal statistic, capturing worst-case rather than average deviation.

The authors validate AFL against a logprob-derived coarsened-KL comparator across three route conditions where logprobs were available from the target API, and find strong linear descriptive agreement between the two. They then run 20-run sequence probes across seven route snapshots and find that EFL varies by route. Neither AFL nor EFL shows much detectable association with accuracy on GPQA-Diamond, a benchmark of short factual and reasoning questions. Pronounced EFL, however, coincides with a decline in Terminal-Bench pass rate as the length of task exposure increases, a pattern the authors suggest may occur because correctness on long-horizon tasks is more sensitive to extreme fidelity loss than short-task accuracy is.

The authors conclude that AFL and EFL should be reported jointly rather than relying on either alone, particularly when auditing long-horizon agentic tasks, where the API-fidelity gap appears to bite hardest. An open-source implementation of Ventor-QTest is available on GitHub, published under Tencent's AI-Infra-Guard project at services/api_checker/ventor_qtest, letting anyone run the audit against a live vendor API without needing the provider's cooperation.

Key facts

  • Ventor-QTest is a composite black-box audit that checks vendor-hosted LLM API quality without needing probability information from the target API.
  • Its repeated-request component computes average fidelity loss (AFL), a null-bias-corrected, within-window mean coarsened-KL statistic, from a frozen prompt sent to the API repeatedly.
  • Its long-sequence component computes extreme fidelity loss (EFL), the empirical upper tail of a reference-centered-surprisal statistic, from independent long-sequence runs.
  • Across three logprob-capable route conditions AFL closely tracks a logprob-derived coarsened-KL comparator; across seven route snapshots with 20-run probes, EFL varies by route.
  • Neither AFL nor EFL correlates much with GPQA-Diamond accuracy, but pronounced EFL coincides with a falling Terminal-Bench pass rate as task exposure increases.

Why it matters

Third-party providers that host open-weight models are now a core part of the LLM supply chain, but there has been no standard way to check whether the inference API they serve actually matches the model and quality it claims, since customers of a hosted API don't normally get the token probabilities needed for a rigorous check. Ventor-QTest addresses that gap using only the text a vendor's API returns.

Who it affects

Anyone building on top of vendor-hosted open-weight models, especially teams running long-horizon agentic workloads, where the paper finds that fidelity loss has a measurable practical effect on task success.

How to use it

The method produces two numbers. AFL comes from sending the same frozen, constrained prompt to the target API repeatedly and comparing the resulting output-text distribution against a reference. EFL comes from independent long-sequence runs and looks at the worst-case tail of a surprisal statistic rather than the average. The authors recommend reporting both together, particularly for long-horizon agentic tasks. An open-source implementation is published on GitHub under Tencent's AI-Infra-Guard project (services/api_checker/ventor_qtest), so the audit can be run against a live API without needing the provider's cooperation.

How solid is it

AFL was validated against a logprob-derived coarsened-KL comparator across three route conditions where logprobs were available, showing strong linear descriptive agreement between the two measures. EFL was measured with 20-run sequence probes across seven route snapshots and showed route-specific variation. Both statistics were tested for association with GPQA-Diamond accuracy, a short-task benchmark, and neither showed much of one, while EFL tracked the Terminal-Bench pass-rate decline as tasks got longer.

Risks and caveats

The abstract does not report numeric values for AFL, EFL, or the size of the Terminal-Bench pass-rate decline, so how large a fidelity gap has to be before it matters in practice isn't stated here. It also does not identify which vendors or specific model routes were audited, or name the authors or their affiliation. The link to Terminal-Bench is described as a pattern that may arise from long-horizon correctness being more sensitive to extreme fidelity loss, a hypothesis rather than a demonstrated mechanism.