An Open-Weight Model Just Out-Hacked Claude on a Real Security Benchmark, for About a Sixth of the Cost

TL;DR

The static-analysis company Semgrep ran a head-to-head benchmark on a real security task, finding IDOR (Insecure Direct Object Reference) bugs in actual open-source applications, and the result is the kind of thing that should make you re-read your model contract. GLM 5.2, an open-weight model you can download and run yourself, scored 39% F1 using nothing but a prompt. Claude Code running Opus topped out at 37% (and the newest Opus actually did worse, at 28%). GLM did it at about $0.17 per vulnerability found, roughly one-sixth of frontier cost. Semgrep's own purpose-built pipeline still won at 61%, which is the whole point: the harness mattered more than the model. The post is titled "We have Mythos at home," and given that the US banned Anthropic's Mythos 5 two weeks ago over cyber-misuse fears, the joke writes itself.

What Semgrep Actually Measured

IDOR is one of the most boringly common web vulnerabilities there is. An app exposes an internal identifier (a user ID, an order number) in a request, and forgets to check whether the person asking is actually allowed to see it. Change /invoice/1041 to /invoice/1042 and you are reading someone else's invoice. It is dull, it is everywhere, and it is genuinely hard for a scanner to catch because the bug is about authorization logic, not a syntactic pattern. You cannot grep for "forgot to check permissions."

That difficulty is exactly why Semgrep picked it. They wanted to know how much of an AI's vulnerability-hunting skill comes from the model itself, and how much comes from the scaffolding wrapped around it. So they ran a spread of models against IDOR detection in real, open-source codebases and scored them on F1, the balance between catching real bugs and not crying wolf on false ones.

The Result That Should Make You Look Twice

Read the chart and two things jump out. First, GLM 5.2 beat Claude Code, not by a rounding error against a crippled baseline, but against Anthropic's own coding agent running Opus. Second, and this is the part that stings, the newer Opus (4.8) scored worse here than the older 4.6. On this particular task, the upgrade was a downgrade. Benchmarks on narrow tasks are noisy and you should not read a model's whole worth off one F1 number, but "the free model you can run on your own boxes won" is not a result you can wave away.

The economics make it worse for the incumbents. Semgrep clocked GLM 5.2 at roughly $0.17 per vulnerability found, about one-sixth of what comparable frontier models cost to do the same work. For a security team running thousands of scans, that is the difference between a line item and a budget meeting.

The one thing GLM did not beat was Semgrep's own multimodal pipeline, which hit 61% with GPT-5.5 and 53% with Opus 4.8. Which brings us to the actual lesson, the one Semgrep buried under the meme.

The Harness Won, Not the Model

Here is the line that matters, straight from the writeup:

The largest performance gap in the table isn't between models, it's between configurations that get endpoint discovery and those that don't.

The open-weight models all ran in a minimal Pydantic AI harness with a single prompt. Semgrep's winning pipeline added endpoint discovery, code filtering, and other plumbing that feeds the model the right context before it ever reasons about a bug. That scaffolding, not a fancier model, is what bought the jump from 39% to 61%.

If you have been reading this blog, that should sound familiar. It is the same conclusion Google reached in its SDLC paper: your agent is roughly 10% model and 90% harness. Semgrep just put a dollar figure and an F1 score on it. The model is the engine; the harness is the entire rest of the car, and a Honda engine in a real chassis beats a Ferrari engine bolted to a skateboard.

Why the Timing Is the Story

Strip away the benchmark and you are left with a policy embarrassment. Two weeks ago the US Commerce Department pulled Anthropic's Mythos 5, its strongest model, off the open market, citing the risk that its safeguards could be jailbroken and the model turned to cyberattacks. Access is now an allowlist. The premise of the ban is that frontier cyber capability is dangerous enough to gate behind the government.

Meanwhile, an open-weight model from Z.ai that anyone on earth can download, with no export control standing in the way, just outscored a US frontier coding agent at finding security holes. You do not need an allowlist to run GLM 5.2. You need a hard drive. Semgrep's title, "We have Mythos at home," is the dry version of a serious point: gating the frontier does nothing about cyber capability when the open-weight field is already a stone's throw behind and shipping freely.

To be fair to the policymakers, "finds IDOR bugs in open-source code" is a defensive task as much as an offensive one, and 39% F1 is a long way from autonomous hacking. But the export logic does not distinguish; it assumes capability is the scarce, controllable thing. This benchmark is a small, concrete data point that it is not.

What a Builder Should Take From This

Do not over-rotate on a single benchmark. F1 on IDOR in open-source repos is one narrow slice, the sample sizes in these studies are small, and your codebase is not their codebase. GLM 5.2 winning here does not mean it wins everywhere, and it visibly lost to a real pipeline.

But the actionable takeaways are solid:

• Try the cheap open model on your security tasks before you assume you need a frontier API. At one-sixth the cost and competitive accuracy, GLM 5.2 is worth a bake-off, especially for high-volume scanning where token cost compounds.
• Spend your effort on the harness, not the model swap. Endpoint discovery and context-shaping bought more than any model upgrade in this test. If your AI security tooling is plateauing, the fix is probably plumbing, not a bigger brain.
• Treat "we banned the dangerous model" as the security theater it partly is. If your threat model assumes adversaries cannot get capable models, update it. They can, for free, today.

Key Takeaways

• The headline: in Semgrep's IDOR benchmark, open-weight GLM 5.2 scored 39% F1 on a bare prompt, beating Claude Code's best config (Opus 4.6 at 37%, and Opus 4.8 at just 28%).
• The cost: GLM 5.2 found bugs at roughly $0.17 each, about one-sixth of comparable frontier-model cost.
• The real lesson: Semgrep's own scaffolded pipeline still won at 61%, and the biggest gap in the data was harness versus harness, not model versus model.
• The irony: this dropped two weeks after the US gated Anthropic's Mythos 5 over cyber-misuse fears, while a freely downloadable model quietly out-hacked a frontier agent.
• The builder move: bake off the cheap open model on your security workloads, and invest in context plumbing over model swaps.
• The caveat: one narrow task, small samples. A signal, not a verdict on either model.

Sources: Semgrep, "We have Mythos at Home", Developers Digest, TechTimes, OWASP: Broken Access Control