Claude Opus 4.7 stopped the supplied exploit in 90.2% of a new study’s vulnerability cases, but only 60.7% of its patches passed tests written by upstream developers. The 29.5-percentage-point gap is the story: suppressing one crash does not establish that a security fix preserves the behavior maintainers intended.
Northwestern researchers Dang K Le, Wenxuan Shi and Xinyu Xing reported the benchmark-reliability study on October 7. It covers 112 historical bugs across 84 open-source projects in C, C++, Go and Rust. The findings concern specific Claude-and-agent configurations in a preprint, not a universal failure rate for all autonomous patching or production security work.
Better models still leave a grading gap
The controlled runs used OpenCode with Claude Sonnet 4.0, Sonnet 4.5 and Opus 4.7, changing the model while retaining the cases. Agents received build and test instructions but initially not the proof-of-concept body. A PoC is the input or procedure demonstrating a reported failure.
Sonnet 4.0 passed the PoC in 77.7% of cases and developer tests in 43.8%. Sonnet 4.5 reached 84.8% and 48.2%; Opus 4.7 reached 90.2% and 60.7%. Opus genuinely improved both measures. Yet among its PoC-passing patches, 32.7% failed developer tests—roughly one apparent success in three, as the public grading analysis emphasized.
Opus also passed existing regression tests in 94.6% of cases. That does not resolve the lower developer-test score. Old suites check established examples, while a test added with the real fix may encode the missing behavior around the newly discovered defect.
The researchers reconstructed unit or integration tests introduced in upstream repair commits and applied them after agent patching. This tests behavior, not textual similarity to the historical patch; a different implementation can still pass. Full resolution required four successive gates: build, repeated PoC suppression, existing regression tests and the added developer test.
PHP shows how a crash can disappear for the wrong reason
A historical PHP bug in range() misclassified mixed input types and could access a non-string as though it were a string. Rejecting mixed types immediately could stop the crashing input and leave many old tests green, but would also reject valid numeric-and-string combinations PHP was meant to coerce.
The upstream repair corrected the classification while retaining supported behavior. A developer test using a nearby valid input distinguished that repair from the restrictive workaround. A crash-only grader would reward both equally.
Such a guard can be a useful emergency mitigation without being a complete repair. Test-suite overfitting is familiar in automated program repair: a patch satisfies observable examples rather than the underlying requirement. In security, it may suppress a symptom, leave another path open and create false assurance that reduces review.
Maintainer tests are a better proxy, not total maintainer acceptance. They do not exhaust performance, compatibility, readability or architecture. A defensible alternative might deliberately change undocumented behavior, so a failed historical test indicates misalignment requiring inspection, not automatically a dangerous patch.
Instructions change the measured capability
With detailed build and test commands, Sonnet 4.0 passed 77.7% of PoCs, versus 74.1% without them. Developer-test success moved from 41.1% to 43.8%. In C and C++ specifically, instructions raised PoC success from 72% to 92%, while developer-test performance moved only from 28% to 36%.
Documentation is legitimate assistance; human engineers benefit too. But the result means two benchmarks cannot safely compare scores without exposing their information budgets. Go and Rust often offer predictable commands, while C and C++ use varied build systems, flags and wrappers. Setup help can change the score before repair quality is meaningfully assessed.
PoC format matters separately. In 62 Go cases, showing source-level PoCs raised crash-test success from 69.4% to 95.2%, while developer tests improved from 41.9% to 48.4%. A readable harness reveals APIs and call structure, making the visible test easier to satisfy.
In 25 C/C++ cases, exposing opaque binary or text input changed neither the 92% PoC result nor the 36% developer result. Both setups can claim to provide a PoC, but one reveals executable intent while the other leaves the input’s meaning to be inferred. That is not an equivalent evaluation budget.
Search and infrastructure can distort scores in both directions
Historical tasks often have public issues, advisories and fixes. The paper documents an agent using web tools to find an upstream patch, and another finding a prior repair in unsanitized Git history. The patches succeeded, but did not cleanly measure independent repair reasoning.
Searching the web and repository history is sensible production engineering. It is contamination only relative to a benchmark claiming unaided reasoning. A retrieval-and-tool-use test can permit it if the claimed capability is explicit. Potential memorization of historical fixes also limits interpretation of absolute scores.
Framework errors can instead hide valid work or expose hidden answers. Permissions, timeouts and test-file collisions can prevent builds, punish slow projects or mark good patches wrong. Individual framework problems affected roughly 5–32% of cases in complete runs, sometimes exceeding leaderboard gaps.
A score therefore belongs to the model, agent scaffold, prompt, tools, source packaging, limits and grading pipeline together. Controlled runs used OpenCode with Claude; case studies included Claude Code with Sonnet 4.5. Those scaffolds should not be treated as interchangeable.
Related studies show why success needs multiple tests
PVBench examined 209 cases across 20 projects and found that more than 40% of patches accepted by basic checks failed additional tests derived from developer fixes. Its dataset and design differ, so it supports the concern rather than supplying a directly comparable Claude rate.
PatchBench found PoC-only validation inflated solve rates by an average factor of 1.83 across 11 agents. Its security replay, unit tests and output checks reduced the strongest agents from above 97% on original PoCs to roughly half the tasks under combined validation.
PatchBench also selected vulnerabilities whose fixes lay outside the crash stack and transplanted them into changed repository contexts to reduce historical-patch recall. That mechanism targets memorization rather than assuming every successful patch demonstrates new reasoning.
Berkeley’s CyberGym-E2E covers 920 vulnerabilities in 139 open-source projects and combines discovery, reproduction and repair. The best configurations repaired about 80% when given the target vulnerability; end-to-end results fell when discovery was required. Shallow guards around the reported crash still appeared.
These studies use different models, tasks and definitions and do not form one ranking. The Cyber Index methodology similarly widens the workflow to auditing, discovery, reproduction and patching without breaking legitimate behavior. The common issue is what a successful percentage actually includes.
A stronger oracle is not the same as production proof
The Northwestern sample required reproducible builds and meaningful tests in upstream commits. Its 112 bugs are varied, but not representative of every production vulnerability. Public historical material may have entered training. Those limits affect absolute rates, while controlled changes to prompts or grading still demonstrate benchmark sensitivity.
Production can be safer with project-specific tests, static analysis, fuzzing, code review and staged rollout. It can also be harder because tests are incomplete, behavior undocumented and dependencies unavailable. Neither the curated benchmark nor a passed PoC alone establishes unattended deployment readiness.
The next useful evidence would report PoC success alongside developer-intent tests, disclose tool and network permissions, sanitized history, timeouts and per-stage failures, and track whether maintainers accept, revise or reject production candidates. Alternative exploit paths and newly introduced weaknesses matter too. Textually matching the old patch would be the wrong replacement for behavioral validation.
A searchable engineering knowledge base can preserve prompts, tool traces, patches and validation results for cross-project learning. It does not make the patch correct. The study’s supported conclusion is that Opus 4.7 improved materially, while its crash-only score overstated developer-aligned success by 29.5 points. A generated repair remains a candidate whose security property and legitimate behavior need independent evidence.