OpenAI Built a Biology Benchmark Its Own Best Model Fails 71% of the Time

TL;DR

OpenAI shipped GeneBench-Pro this week, a research-grade benchmark that drops an AI agent into 129 messy computational-biology problems and asks not "what is the answer" but "what would a good scientist do next." Each problem is built from a synthetic dataset whose true causal structure is fully known, so the grading is deterministic and leak-proof. External reviewers estimate a single task would eat 20 to 40 hours of an expert's time and cost thousands of dollars. OpenAI's own strongest model, GPT-5.6 Sol, passes 28.7 percent of them (31.5 percent in Pro mode). That is a striking result: the same model class that now clears roughly 80 percent of SWE-bench coding tasks folds when the job is judgment under ambiguity rather than recall or code.

What GeneBench-Pro actually measures

Most benchmarks reward a model for knowing things: facts, formulas, the right function call. GeneBench-Pro is built to reward something OpenAI calls "research taste," the sequence of judgment calls that sits between a dirty dataset and a defensible conclusion. Can the agent look at a real-world-shaped genomics table, notice that the measurement is biased or a quality-control step failed, pick the right analysis strategy over three plausible-but-wrong ones, and decide whether its own finding is robust enough to build on?

The mental picture: this is the gap between the medical student who aced every written exam and the diagnostician who can stare at a contradictory chart and know which test to order next. GeneBench-Pro is the second exam, and it is open-book. The models have read every textbook. They still cannot reliably choose the right chapter.

The scores, told straight

GPT-5.6 Sol, at its highest reasoning setting, passes 28.7 percent of the 129 problems, rising to 31.5 percent in Pro mode. For contrast, plain GPT-5 scored under 5 percent on the original, easier GeneBench. OpenAI reports that GPT-5.6 Sol solves roughly six times as many questions as GPT-5.2 while spending about two-thirds the tokens, so the reasoning models are climbing fast. They are just climbing from a very low floor toward a still-failing grade.

Why the design is the clever part

The load-bearing trick is that every dataset is synthetic, generated from a known data-generating process. That buys two things a scraped real-world benchmark cannot. First, the ground truth is exact, so grading is deterministic rather than a matter of an LLM judge squinting at prose. Second, because the data never existed before, the answers cannot be sitting in anyone's training set. OpenAI says it also sent 82 of the problems to external domain experts for review and audited drafts for information leakage, which is the unglamorous plumbing that separates a benchmark you can trust from a leaderboard you cannot.

The problems span 10 domains and 21 sub-domains, from statistical and population genetics to cancer genomics, clinical diagnostics, and pharmacogenomics. This is not trivia. It is the daily judgment work of a computational biologist, priced by reviewers at 20 to 40 hours and thousands of dollars per task, against a few dollars of inference for the model. The economics are absurd in the model's favor. The pass rate is absurd in the other direction.

Why a builder should care

• Recall is solved, judgment is not. If your agent's job is retrieval, summarization, or writing code against a clear spec, current models are strong. If its job is choosing among defensible analysis paths on dirty data, GeneBench-Pro is your warning label: expect a failure rate near two thirds, and design for it.
• Deterministic, leak-proof evals are the gold standard. The synthetic-with-known-ground-truth pattern is worth stealing. If you can generate your eval data from a process you control, you get exact grading and immunity to contamination in one move, no LLM judge required.
• "It costs a few dollars" is not "it is reliable." The gap between inference cost and expert cost is enormous and seductive. On judgment-heavy scientific work, cheap and wrong is still wrong, and a confidently wrong conclusion in genomics is more expensive than no conclusion.
• Watch the trajectory, not just the number. Six times the score of GPT-5.2 at two-thirds the tokens is a real slope. The floor is low today; the derivative is not. Plan for the domain where your agent fails now to be the one it half-passes in two model generations.

Key Takeaways

• GeneBench-Pro is OpenAI's new benchmark of 129 synthetic computational-biology problems across 10 domains, testing scientific judgment rather than factual recall.
• Its own best model, GPT-5.6 Sol, passes 28.7 percent (31.5 percent in Pro mode); plain GPT-5 scored under 5 percent on the earlier GeneBench.
• Each task is estimated at 20 to 40 hours of expert time and thousands of dollars, versus a few dollars of model inference.
• Synthetic data with fully known causal structure makes grading deterministic and immune to training-set contamination, a pattern worth copying for your own evals.
• Coding benchmarks are near saturation while judgment-under-ambiguity is not, so agents built for messy analytical work need failure-tolerant design today.

Sources: OpenAI, Digital Watch Observatory, Let's Data Science