Alibaba's Co-Design Bench Finds Coding Agents Fully Finish 26.4% of Real Tasks

TL;DR

Co-Design Bench, an Alibaba-led benchmark built with five universities, grades coding agents on 273 revision tasks recovered from 26,314 real user coding sessions, against 2,331 acceptance criteria the agent never sees. Under one fixed harness, the best of ten model endpoints, GPT-6 Astra, passes 69.9% of criteria per task but fully finishes only 26.4% of tasks. The agent usually does most of the work, drops one requirement, and leaves the last mile to you. Swap the harness and keep the model, and scores move by nearly 19 points: a leaderboard row is measuring two products, not one.

Built from sessions that already went wrong

The team, from Alibaba's Token Hub with Fudan, Zhejiang, CUHK, SJTU and NTU, started from 39,325 production queries across 26,314 real coding sessions, deduplicated down to 37,463, then filtered and ranked candidates by difficulty, verifiability and historical failure signals. In plain terms, they favored requests that had already made agents stumble: the rephrased-three-times, rolled-back-twice kind of ask. At most three queries per session survived, and experts consolidated each into one bounded task with hidden criteria.

Every starting application had to build, launch and render before it qualified, so the agent inherits a real, slightly messy codebase instead of a toy.

Two tracks split the work. Multimodal Editing (ME) is 100 localized edits, sometimes grounded in an image, an audio recording, a video clip or a document, like pulling a total off a receipt and updating every dependent view. Repository Editing (RE) is 173 coordinated multi-file changes. RE tasks average 10.3 criteria each, and 166 of the 173 need two or more types of change at once.

The scoreboard

All ten endpoints ran in the same Claude Code harness, with container manifests, system prompts and tool interfaces held fixed, capped at three hours and 120 turns per run. GPT-6 Astra leads. Anthropic's Opus 5 and Qwen3.8-Max take the next spots, Grok 4.6 closes the table, and the paper shipped with code, an evaluation pipeline and 96 ME tasks published on Hugging Face.

69.9% of the rubric, 26.4% of the jobs

Passing 69.9% of criteria per task sounds respectable until you notice only 26.4% of tasks get every criterion passed. Partial credit earns nothing in strict scoring, so the typical outcome is an agent that nails most requirements and misses one: the button that works, except in dark mode, for the state you did not test.

Grading is adversarial on purpose. The finished app is built and launched in a sandbox, evidence is gathered per criterion from source, screenshots and browser interactions, a judge issues PASS, PARTIAL or FAIL, and then a separate verifier challenges every positive verdict, hunting for stubs, hardcoded values and unreachable features. A visual claim with no render or screenshot aborts the evaluation. Picture an oral exam where every answer you got right gets re-checked to confirm you solved it rather than wrote the answer on your palm.

Compound requests are where agents die

Across all ten endpoints, repository edits spanning three or four change types (structure, style, interaction, content) score 6.3 points lower on strict completion than edits spanning one or two, and zero-pass rates climb from 0.9% to 5.8%. Real requests are compound by default; nobody asks an agent to only change the spacing. The benchmark's own numbers put the fix on your side of the keyboard. Split the compound ask into a checklist and you are converting RE-shaped requests into the narrow kind agents finish.

The harness is half the product

For the main table the harness was fixed, which is exactly why the appendix matters. The authors ran two solvers, Kimi K3 and Qwen3.8-27B, through three harnesses: codex, opencode and qwen-code. Same model, same 273 tasks, same criteria. Qwen3.8-27B completes 56.0% of RE criteria under opencode and 74.9% under qwen-code. On RE, the Kimi-versus-Qwen gap runs from Kimi ahead by 12.2 points to Qwen ahead by 4.0, and the ordering flips with the environment. On ME the harness ranking inverts outright between the two solvers.

When you pick a coding agent you are buying a model-harness pair, and either half can be the weak one. Test the pair. A vendor quoting a leaderboard position without naming the harness is quoting half a result.

Effort is not completion

The token counters undercut the just-let-it-grind theory. On ME tasks, GLM-5.2 averages 69 turns and 7.29M tokens per task for 57.2% strict completion. GPT-6 Astra averages 32 turns and 1.24M tokens for 75.4%. Nearly six times the thrash for 18 points less.

Why your agent aces every public benchmark

The saturation check is blunt. On 150 tasks from an existing web-editing benchmark, switching a fixed model from single-shot to agentic execution raised graded completion from 73.6% to 95.1%, and Opus 4.8 hit 99.4%. Once an agent can run, check and retry its own code, most web-coding benchmarks stop measuring the model; they measure the loop. Co-Design Bench was built to survive that: real sessions, hidden criteria, compound edits, and a refutation pass for fake wins.

One detail worth stealing for your own evals: agents here improvise their own perception stack, calling OpenCV in 68.5% of video runs and ffmpeg in 31.5%, and some pull Whisper, CLIP or DINOv2 mid-task to ground an edit. Your coding agent is already a systems integrator.

What to do Monday

• Write acceptance criteria into your own tickets. The gap between 69.9% and 26.4% is the gap between looks done and passes the rubric. A five-line checklist per task converts vibes into pass or fail.
• Split compound requests. Edits spanning three or four change types score 6.3 points worse and zero out six times more often. Sequence them instead of stacking them.
• Test model and harness as a pair before you buy. The same model moved almost 19 points between harnesses on identical tasks.
• Cap the grinding. Nearly six times the tokens bought 18 fewer points in this data. A turn budget controls quality as much as cost.
• Discount saturated leaderboards. If the harness can retry, a 95-plus score tells you about the loop, not the model.

Caveats

• The paper landed on arXiv October 26 and the repository went public October 28; the coverage wave is this weekend. New to most readers, not to the calendar.
• The task pool deliberately favors hard cases, ranked for difficulty and past failure. 26.4% is a floor on adversarially selected work, not an industry-wide success rate.
• Grading is LLM-judged, with a refutation pass and evaluator-robustness checks in the appendix, but judges can err in both directions.
• One request per task, no follow-ups, no mid-task corrections. Real co-design is conversational; this measures the single-shot slice.
• The main table holds one harness fixed. Your harness is different, and the appendix says that gap is worth double digits.

Key Takeaways

• Co-Design Bench grades 273 tasks recovered from 26,314 real coding sessions against 2,331 hidden expert criteria, in two tracks: 100 multimodal edits and 173 repository revisions.
• Best of ten endpoints (GPT-6 Astra, fixed Claude Code harness): 69.9% of criteria passed per task, but only 26.4% of tasks fully delivered. Opus 5 and Qwen3.8-Max follow; Grok 4.6 is last.
• Compound requests are the failure mode: edits spanning three or four change types score 6.3 points lower, and zero-pass rates rise from 0.9% to 5.8%.
• The harness swings results as much as the model: Qwen3.8-27B scored 56.0% to 74.9% on the RE track across three harnesses, and the model ranking flipped with the environment.
• Effort did not pay: GLM-5.2 burned 7.29M tokens per ME task for 57.2% strict completion, versus GPT-6 Astra at 1.24M tokens for 75.4%.
• On an older benchmark, agentic execution pushed a fixed model from 73.6% to 95.1% completion, so saturated leaderboards measure the retry loop more than the model.

Sources: Co-Design Bench project site and leaderboard, arXiv 2610.35138, GitHub repository, Hugging Face dataset, The Decoder