A New Benchmark Gave AI Real Office Work. The Best Model Fully Nailed 3% of It.

TL;DR

Artificial Analysis just shipped AA-Briefcase, a benchmark that drops a model into four multi-week corporate projects and asks it to produce real deliverables (spreadsheets, memos, decks) by digging through nearly 2,000 source files. The headline number is brutal: the top model, Claude Fable 5, fully satisfied every requirement on only 3% of the 91 tasks. On 31 of those 91 tasks, no model cleared a 50% pass rate. If you are building agents to do knowledge work, this is the cold-water benchmark you needed.

What AA-Briefcase actually tests

Most benchmarks you have seen lately are clean: a coding task with a unit test, a math problem with a single answer. AA-Briefcase is the opposite. It simulates the messy reality of an analyst's job. Each of the four held-out scenarios is a multi-week project assembled with industry experts from data science, product management, banking operations, and heavy-industry strategy.

The catch is the context. To do the work, a model has to wade through the kind of fragmented junk a real employee inherits:

• Nearly 2,000 source files per the full set
• 3,500+ emails and 25,000+ Slack messages
• Meeting transcripts, spreadsheets, and large raw data exports

The output is not a tidy answer. It is a deliverable: a financial model, a strategy memo, a slide deck. The benchmark grades each of its 91 tasks on three axes: a binary rubric (did you get the objective facts right), an analytical-quality Elo from pairwise comparison, and a presentation-quality Elo. Grading is decided by a panel of three judges (Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro) rather than a single model, to blunt single-judge bias.

The leaderboard, and why it is sobering

Claude Fable 5 wins the overall Elo, but "wins" needs an asterisk. Here is the verified overall Elo board:

GLM-5.2 is the best open-weight showing at 1261, slotting between GPT-5.5 and the Anthropic models, which is a genuinely good result for a model you can self-host. DeepSeek V4 Pro sits well back, and the Gemini 3.1 Pro preview lands at the bottom, a reminder that long-horizon, multi-file agentic work punishes harnesses that are not tuned for it.

Nobody is actually good at this yet

The Elo ranking hides the real story, which is the absolute pass rate. Even the winner is failing most of the time.

That gap matters for how the models fail. Artificial Analysis found the failure modes split by capability. Weaker models break on the basics: they cannot retrieve the right file or execute the steps. Stronger models fail more quietly. As the team put it:

Stronger models fail more quietly, as they hit the obvious requirements but miss details you'd only catch by piecing together information from multiple sources.

That is the part that should worry anyone shipping an agent into production. A model that nails the obvious checks and silently drops the cross-source detail is exactly the kind of failure a human reviewer waves through.

The cost spread is wild

One more number for the builders watching unit economics: per-task cost across the field spans more than 800x. Claude Fable 5 ran over $31 per task; DeepSeek V4 Flash came in around $0.04. The frontier score is real, but so is the bill, and on a benchmark where the ceiling is 3% you should think hard about whether the top-tier model is worth roughly 800 times the cheapest option for your workflow.

Key Takeaways

• AA-Briefcase is the realism check the field needed: 91 tasks across four multi-week projects, graded on rubric, analysis, and presentation by a three-model judge panel.
• Everyone is bad at real knowledge work. The best model fully satisfies all criteria on just 3% of tasks; on 31 of 91, no model beats 50%.
• Claude Fable 5 leads at 1586 Elo, ahead of Opus 4.8 (1353) and GPT-5.5 (1159).
• GLM-5.2 (1261) is the open-weight standout, beating GPT-5.5 and trailing only the two Anthropic models.
• Stronger models fail silently, missing cross-source details rather than fumbling the basics, which is the harder failure to catch in review.
• Cost spans 800x (about $31 vs $0.04 per task), so the top score is not always the right call for your budget.

Sources: Artificial Analysis: Announcing AA-Briefcase, AA-Briefcase evaluation leaderboard, The Decoder