GenV verifier catches proof translations that fool solvers

Neurosymbolic systems lean on mathematical solvers to guarantee that a piece of reasoning is correct, but the authors of this paper point out a blind spot: a solver only checks whether a formal translation produces the expected outcome, not whether that translation is actually a faithful, reference-equivalent copy of the thing it was supposed to encode. An incorrect encoding can still run successfully and land on the correct verdict anyway. The authors name this failure mode Verdict-Preserving-Unfaithfulness (VPU) and prove theoretically that structural, verdict-only verification heuristics are mathematically capped at chance-level detection on these deceptively valid traces: no amount of tuning a check that only looks at the final verdict can reliably catch them.

To fix the gap, the paper introduces Generative Verification (GenV), which distills an offline Z3-equivalence oracle into a continuous reference-equivalence score. Instead of needing the original reference at inference time, GenV repurposes the language model's own vocabulary space to produce that score directly. The authors also run a mechanistic analysis, using decision-projected logit lenses and sparse autoencoders, and find that this generative readout natively picks out the precise spatial location of an error inside a translation, despite never being explicitly trained to localize errors.

On the empirical side, their oracle-mined verifier, called GenV+HN, reaches 0.961 AUROC at reference-equivalence verification and generalizes zero-shot to translators and formal styles it was not trained on. Used to guide agentic test-time compute allocation, GenV+HN produces an 11.3-point downstream accuracy gain. The abstract does not name the paper's authors, institution, publication venue, or review status, does not specify which model, dataset, or task domain produced the 0.961 AUROC and 11.3-point figures beyond the terms already used in the abstract, and does not expand the 'HN' abbreviation in GenV+HN.

Key facts

Solvers check only whether a formal translation lands on the expected verdict, not whether the translation is a faithful, reference-equivalent copy; the authors name this blind spot Verdict-Preserving-Unfaithfulness (VPU).
The paper proves that structural, verdict-only verification heuristics are mathematically bounded to chance-level detection on these deceptively valid but incorrect translations.
GenV distills an offline Z3-equivalence oracle into a continuous reference-equivalence score by repurposing the language model's own vocabulary space, so it does not need the reference at inference time.
The oracle-mined verifier GenV+HN reaches 0.961 AUROC in reference-equivalence verification and generalizes zero-shot to unseen translators and divergent formal styles.
Mechanistic analysis shows the generative readout extracts precise error locations without explicit localization training, and GenV+HN yields an 11.3-point downstream accuracy gain in agentic test-time compute allocation.

Why it matters

Neurosymbolic systems are trusted precisely because a solver's approval is supposed to mean the reasoning was verified. This paper shows that trust can be misplaced: a translation into formal terms can be wrong and still pass, because the solver only checks the final verdict rather than whether the translation faithfully preserves the original meaning. That gap, which the authors call Verdict-Preserving-Unfaithfulness, undercuts the core guarantee that makes solver-backed pipelines attractive in the first place, and the paper's proof that verdict-only checks are capped at chance-level detection means the fix has to come from a different kind of verifier, not a better-tuned version of the same one.

Who it affects

Researchers and engineers building autoformalization pipelines, meaning systems that translate natural-language statements or reasoning into formal, machine-checkable representations, are the direct audience. It also matters for anyone building agentic systems that allocate test-time compute based on verification signals, since the paper reports a downstream accuracy gain from using GenV+HN in exactly that setting.

How to use it

GenV is presented as a verification method, not a shipped product: it distills an offline Z3-equivalence oracle into a reference-free, continuous score that a system can compute using a language model's own vocabulary space, without needing the original reference translation at inference time. The abstract does not describe a release, code, dataset, or specific integration path beyond that description.

How solid is it

The claim rests on two legs: a theoretical proof that verdict-only structural heuristics cannot beat chance-level detection on deceptively valid traces, and empirical results showing GenV+HN reaches 0.961 AUROC and generalizes zero-shot across translators and formal styles it was not trained on. The mechanistic-interpretability analysis, using decision-projected logit lenses and sparse autoencoders, adds a second line of evidence for the localization claim. The abstract does not say which model, dataset, or task domain produced the 0.961 AUROC or 11.3-point figures beyond the terms it already uses, so how representative those numbers are of other settings is not stated.

Risks and caveats

The abstract names no authors, institution, publication venue, or review status, so this cannot yet be checked against peer review or an author's track record. It does not expand or explain the 'HN' abbreviation in GenV+HN, and it gives no detail on the model, dataset, or task domain behind the headline numbers beyond the terms already stated in the abstract itself.