Hightouch's Cheap-Model Judge Matches Claude Opus at a Quarter of the Cost

TL;DR

Hightouch published a benchmark on November 4 for a chore data teams grind through in silence: checking whether an identity resolution system has merged customer records into the right human beings. A cheap decision model called Jev judges 1,000 identities for about five cents. Escalate the 22% of cases where Jev wavers to Claude Opus 5.5 and the pair scores 98.8% accuracy at $0.88 per 1,000, against 99.0% at $3.86 for Opus grading everything. On a deliberately harder test set the hybrid passed Opus outright, 95.3% to 95.1%, for a bit over half the spend. The whole design hangs on one property: calibrated confidence.

The chore: auditing identity resolution

Identity resolution maps warehouse records onto real people, so the loyalty signup, the two transactions and the newsletter bounce become one customer. The failure mode that ruins your numbers is over-merging, where a merged profile absorbs a parent and a child, or two strangers who share a family Gmail address.

Real data has no answer key, so teams grade samples by hand. Hightouch puts human review at roughly 50 identities an hour, and because over-merges are rare in a healthy graph, you need more than 1,000 samples before the error bar on your over-merge rate drops under a point. A model that could do the same read at human accuracy would compress a month of sampling into an afternoon, which is why the team ran the experiment.

The setup

The benchmark uses pseudopeople, the University of Washington project that simulates an entire US town's administrative records. From it the team built 1,000 test identities, then injected realistic noise: typos, misread characters, a birth date copied from a relative, a swapped month and day. Half the identities are clean and half are planted over-merges. Three frontier models ran the gauntlet, Claude Opus 5.5 among them, against two decision models: Jev, and GPT-6 Luna through OpenAI's Decisions API.

Decision models do not write prose. You hand them evidence and a condition, they hand back a probability between 0 and 1 in around 150 milliseconds. An if-statement you rent by the microsecond.

Cheap judges hold up until the data gets mean

On the first pass the cheap models look respectable. Opus graded 99.0% of identities correctly at $3.86 per 1,000, taking 2.2 seconds each; Gemini 3.1 Pro and GPT-6 Sol landed at 98.3% for $4.48 and $2.50. Luna came in at 97.3% for $0.10 and Jev at 95.7% for $0.05, both answering in about a sixth of a second. Hightouch's own summary: decision models are up to 90x cheaper and more than 10x faster out of the box.

Then the team made the test mean. Same town, only hard cases: twins, a junior and a senior at one address, two unrelated people with the same name. The frontier held, with Opus at 95.1%. The cheap models fell apart, Jev to 89.1% and Luna to 88.8%. Six points is six points, and the obvious move looks like buying the frontier model and its invoice.

Calibration is the routing signal

Jev's mistakes cluster near 50% confidence: it is wrong, but it shrugs while being wrong. Luna's mistakes pile up in the 90-100% bucket: it is wrong at the top of its lungs, and on the harder set the volume climbs.

That is the difference between a bad answer and a routable one. A well-calibrated cheap model is a triage nurse: the cases she flags as unclear go to the specialist and the rest never touch his calendar. An overconfident one sends you home with something serious.

Escalate the unsure middle

The hybrid is one rule. Jev grades everything, and any identity where its confidence lands between 30% and 90% gets a second opinion from Opus. On the easy set that band covers 22% of cases, and the pair scores 98.8% at $0.88 per 1,000 with a 0.63 second average. Opus alone is 99.0% at $3.86. Widening the band to 10-90% sends 40% of cases and buys the same 98.8% for $1.58, so the narrow band is where the money lives.

On the harder set, same harness, same thresholds, no retuning: Jev simply finds more cases uncertain, routes more of them to Opus, and the bill flexes from $0.88 to $2.94 per 1,000 on its own. Accuracy lands at 95.3% against 95.1% for Opus grading everything. Same accuracy class, 45% less spend, and a system that spends more only when the data demands it.

Luna got the same harness. Its overconfidence means it escalates almost nothing: 6% of cases on the easy set, 98.5% accuracy at $0.33, the cheapest configuration on the board. On the hard set it sagged to 91.6%. The model that looked better standing alone is the wrong foundation for a cascade.

The model class underneath

Jev comes from TypeSafe AI, founded by Diogo Almeida, who says he worked on the instruction-following research behind ChatGPT during his OpenAI years. TypeSafe calls Jev a System One Model: it gives up string generation entirely, returns typed values with calibrated probabilities, samples its outputs in parallel, and is trained with what the lab calls Reinforcement Learning for Calibrated Decisions. Published rates are $0.042 per million input tokens, output free, responses in 70 to 500 milliseconds. The name honors William Stanley Jevons, the economist who explained why cheaper coal meant more coal, which is either a joke or a business plan.

OpenAI's counterpart is the Decisions API, generally available with GPT-6 Luna as its only model. You ask predicate, choice or score questions, up to 400 per request, and get typed answers about 10x faster than the Responses API. Two vendors selling the same shape of model reads like the start of a category, and Hightouch's numbers are the first public accounting of what it is good for on a messy production task.

The pattern is already loose

The Hacker News thread pulled over 280 points, and the comments read like a rollout plan. A security engineer describes judging tool calls and intent with the same hybrid after an initial calibration pass. Another commenter wired a local decision model into an agent harness to validate tool inputs and outputs, with mixed enthusiasm. Someone else points out that Claude Code ships a native version of the idea: Advisor lets a weaker model call in a stronger one when the case seems to warrant it.

What to steal

• Buy calibration before accuracy. A cheap model that knows when it is unsure is composable; an overconfident one sends its mistakes through unchanged.
• Route on a band, not a cliff. Sending the unsure middle (30-90% confidence) matched sending everything below 90% on accuracy at roughly half the cost.
• Escalate to the best model you can rent, not the second cheapest. The specialist only sees what triage cannot, so its share of the bill stays small.
• Recheck the cheap model's calibration on your hardest slice before trusting the routing. Easy data hides overconfidence.

Caveats

• Every number above is Hightouch's own measurement on its own benchmark, published on its blog. No third party has rerun it, and no code shipped with the post; commenters asked for exactly that.
• The data is synthetic. pseudopeople's noise model is careful, but your warehouse's pathologies are your own.
• One task. Grading identity merges may flatter decision models more than judging open-ended writing would.
• Jev is early access from a young lab, and the hybrid's savings assume Opus pricing holds still, which recent price-cutting across providers suggests it will not.

Key Takeaways

• Hightouch benchmarked LLM judges for identity resolution auditing: 1,000 pseudopeople identities, half planted over-merges, graded by three frontier models and two decision models.
• Out of the box, Jev costs $0.05 per 1,000 identities and Luna $0.10, against $3.86 for Opus at 99.0% accuracy; the cheap models trail by roughly 2 to 6 points.
• The winning rule: Jev grades everything, and cases landing between 30% and 90% confidence go to Opus. That is 22% of cases, 98.8% accuracy, $0.88 per 1,000: Opus-class results at a quarter of Opus's bill.
• On the harder test the untuned harness flexed by itself, routing more cases and scoring 95.3% versus Opus's 95.1% at $5.39 per 1,000.
• Jev beat Luna inside the cascade because its errors sit near 50% confidence while Luna's sit above 90%. Calibration picks your cheap judge, not standalone accuracy.
• The cheap-judge-plus-specialist pattern is already spreading through security harnesses and agent tooling, and Claude Code's Advisor feature ships a native version of it.

Sources: Hightouch engineering post, TypeSafe AI Jev announcement, OpenAI Decisions API docs, pseudopeople, Hacker News thread, Claude Opus 5.5