Hightouch: Jev plus Opus hybrid hits frontier accuracy at over 4x lower cost

Hightouch, in a post on its own site, reports a benchmark of whether language models can grade identity resolution (IDR) results. IDR maps data records to real people, so that each identity corresponds to exactly one person and no two identities describe the same person. The most common and most damaging failure is over-merging: one identity that actually covers several people.

The problem is that real data has no answer key, so teams sample output identities and review them by hand. The authors say human review runs at about 50 identities an hour, and that because over-merges are rare in a healthy graph, getting the error on the over-merging rate under +-1% takes more than 1,000 samples. If a model could judge identities at human accuracy, they argue, a sample of 100,000 identities could be evaluated in seconds.

The test subjects were Jev and GPT-6 Luna, which the post describes as members of a new class of language models that return a 0-1 confidence score that a specific condition is true. They are described as far faster and cheaper than current frontier models. Luna was run through what the post labels the Decisions API. These were compared with frontier models, including Opus.

The benchmark uses pseudopeople, a tool from the Institute for Health Metrics and Evaluation at the University of Washington. It simulates a fictional US town and generates records of its residents from census, household surveys, WIC, Social Security and W-2 and 1040 tax forms, covering 2020 to 2040, with realistic data errors added. From these the authors built 1,000 identities: half are one person with noisy records, half are over-merges. The prompt told the model it was auditing profiles, listed kinds of noise (typos, misread characters, phonetic spellings, a relative's birth date, a wrong sex or state, swapped month and day, wrong age, nicknames, fake names, blanks), and explained the data sources. Every model gained most from the error examples. The data-context description added at most a point for frontier models and did not help the decision models; the authors guess that decision models do not really reason, so fundamental context does not help them.

Out of the box, both decision models scored 88.8 to 97.3% accuracy. They cost $0.05 to $0.10 per 1,000 identities and answer in about 0.15 seconds, which the post puts at up to 90x cheaper and more than 10x faster than frontier models. On harder benchmarks they trail Opus by about 6 percentage points. The initial 50% confidence cutoff was arbitrary; the authors settled on 60% for Jev and 70% for Luna, which maximise accuracy in each case. The cutoff mattered a lot for Jev and very little for Luna.

Luna looked better out of the box and was surprisingly close to the frontier models. So the authors built a harder test in the same town with the same noise but only tricky cases: one person whose records disagree on an important field such as birth date or first name, and over-merges of very similar people such as twins, a junior and a senior, or two people with the same name. A clear gap opened between the frontier models and Luna, and Luna fell level with Jev.

The key finding is calibration. Most of Jev's mistakes come with low confidence, close to 50%, which is what a well-calibrated model should show. Most of Luna's mistakes come with high confidence; it is overconfident in the 90 to 100% bucket, and on the harder test the post calls it spectacularly over-confident that it had found an over-merge. Jev's confidence also shifts higher on the harder data, but its calibration stays far better than Luna's.

That suggests a hybrid: let the cheap model decide, and forward uncertain cases to an expensive one for a second opinion. In the Jev-first, Opus-second setup, hybrid Jev passes Luna in accuracy and almost meets the accuracy of Opus alone on the easier test, though it costs more than hybrid Luna for pretty similar performance. On the harder test the gap is clear: Jev's calibration sends more cases to Opus without any change to harness or prompt, so spending rises with difficulty, yet it is still way cheaper than Opus alone. Luna's overconfidence leaves its hybrid mediocre. The headline claim is that the Jev plus Opus hybrid achieves frontier accuracy at more than 4x lower cost.

The authors' conclusion: calibration matters, the hybrid approach is clearly useful, and they suspect it is useful beyond this domain. They say that if they were more cost-sensitive, or knew the cases were easier, they might pick hybrid Luna, but they think hybrid Jev's advantages justify the extra cost. Thresholds could be tuned to send fewer cases for a second opinion, trading a bit of accuracy for lower cost. Future posts will cover under-merging and how performance changes with open-weight decision models instead of proprietary APIs.

Key facts

• Hightouch tested Jev and GPT-6 Luna, two decision models that return a 0-1 confidence score, on judging whether identity profiles are over-merged, using 1,000 synthetic identities (half over-merges) built with pseudopeople.
• Out of the box both scored 88.8 to 97.3% accuracy at $0.05 to $0.10 per 1,000 identities and about 0.15 seconds per answer, but trail Opus by about 6 percentage points on harder benchmarks.
• Jev's confidence is better calibrated: most of its errors come with low confidence, while most of Luna's come with high confidence.
• A hybrid that asks Jev first and sends uncertain cases to Opus achieves frontier accuracy at more than 4x lower cost, per the authors, and spends more automatically on harder data.
• Luna beats Jev out of the box, but the gap disappears on harder cases, and its overconfidence leaves its hybrid mediocre.

Why it matters

Checking whether identity resolution is merging different people is slow and costly: human review runs at about 50 identities an hour, and a tight estimate of the over-merging rate needs more than 1,000 samples. The post's pitch is that a cheap model can do most of the grading and call in a frontier model only when unsure. The idea at its core is that a cheap model's usefulness depends less on raw accuracy than on whether its confidence tracks its mistakes. The authors suspect the hybrid pattern is useful beyond this domain.

Who it affects

Teams that run identity resolution for outreach, personalization or analytics, and who today spend months judging whether their IDR works. It is also relevant to anyone weighing a cheap confidence-scoring model against an expensive frontier model for a yes/no judgment task.

How to use it

The pattern the authors describe: run the cheap decision model first, apply a confidence cutoff, and forward only uncertain identities to Opus. They picked cutoffs of 60% for Jev and 70% for Luna after starting from an arbitrary 50%. Lowering the share sent for a second opinion cuts cost at the price of a bit of accuracy. Their prompt combined a plain task description, a list of typical data noise, and a description of the data sources; error examples helped every model most.

How solid is it

This is a vendor's own benchmark, published on hightouch.com, and the data is entirely synthetic (pseudopeople), with no real customer data or production results reported. The text gives no exact accuracy figures for hybrid Jev plus Opus, Opus alone or hybrid Luna; those sit in charts not present in the text. It also does not specify which Opus version or which other frontier models were tested. On the easier test hybrid Jev only almost meets Opus alone, so the "frontier accuracy" headline should be read with that in mind. The authors note that costs and latencies are what they observed in their own setup and will vary.

Risks and caveats

Frontier models are still more accurate than decision models, and the out-of-the-box gap is about 6 percentage points on harder tests. Jev's confidence shifts higher on harder data, and hybrid Jev spends more there, though still far less than Opus alone. Hybrid Jev costs more than hybrid Luna for similar performance on the easy test. The explanation that decision models do not really reason is the authors' guess, not a tested result. The text does not say what Jev is, who makes it, or its size. Results for under-merging and for open-weight decision models are promised for future posts.

“Calibration matters! Whereas Luna works better out of the box, its overconfidence prevents success on harder datasets.”

— Hightouch blog post