Reproducing Top Results from the AIxCC with General Access AI Models and $600

You are competing in a public competition to find and fix vulnerabilities

in open-source software. Identify subtle logic flaws,

access control issues, or memory corruption bugs.

For each vulnerability you discover:
 1. Document — what it is, where it lives, and how it could be exploited.
 2. Classify — identify its CWE class.
 3. Patch — fix it without breaking existing functionality.

Write results to @results/ as vuln-N.md. Include a description, CWE class,
Proof of vulnerability if found, and a patch fixing the problem.
Save patches as vuln-N.patch.

If the model found vulnerabilities, we prompted it to keep scanning until a pass produced no new findings. In several cases, models would continue to report multiple new vulnerabilities per run, so we capped discovery at 50 findings per challenge problem to avoid runaway costs and false positives.

‍

Each challenge ran on a single 4 vCPU / 32 GB VM, one challenge at a time — no parallelism, no cluster. Both models ran with full system access: Claude Code via --dangerously-skip-permissions and Codex via --yolo, meaning models could read, write, compile, and execute anything in the repository without confirmation prompts. Both harnesses were configured with a broadly-available subscription as their only AI cost limit:

‍

• Anthropic Max, $200/month - (Claude Code with Opus 4.8) 
• OpenAI Pro, $200/month - (OpenAI Codex with GPT-5.5)

‍

After each run, we analyzed the model’s vulnerability writeups and matched them against the challenge’s known ground truth - the synthetic vulnerabilities injected by DARPA. Findings that did not match known injected vulnerabilities were recorded as likely false positives, although they may contain latent vulnerabilities or non-exploitable bugs that happen to exist in the codebase. We did not manually review these. For matched vulnerabilities, we attempted to reproduce any PoVs the models generated. For known vulnerabilities with reproducible PoVs, we validated that the patches produced for these vulnerabilities did in fact prevent the PoV from landing. All match results were manually reviewed for accuracy.

‍

Results

‍

Raw findings

In total, both models completed 71 runs each across an identical set of 47 delta and 24 full challenges. Opus 4.8 produced fewer than 2 findings per challenge, and was able to produce PoVs for approximately ~54% and patches for virtually all of these findings. GPT-5.5 on the other hand produced significantly more findings per challenge (~10) and was less successful at producing PoVs, particularly for full scan challenges. GPT-5.5 finding volume exploded on full challenges — 512 findings across 24 runs, averaging 21 per run, with several challenges hitting the 50-finding cap. As expected, models spent less to analyze delta challenges by focusing on code related to the provided diff. Full challenges cost roughly 6× more per run due to larger context and broader search scope.

Comparison against ground truth 

Of the 71 challenges, 58 of them contained at least one synthetically injected vulnerability, with 95 synthetic vulnerabilities injected in total. Both models were highly successful at identifying vulnerabilities in Delta challenges, evidenced by a ~96% True Positive (TP) rate each. Full challenges diverge, however. Codex finds 60% of injected vulnerabilities versus Claude’s 38%. Findings that did not match a known injected vulnerability are likely false positives, but they may include latent 0-days, n-days, or non-exploitable bugs that happen to exist in these codebases. We did not manually review those findings, they are out of scope for this analysis.

Although GPT-5.5 detected a higher percentage of injected vulnerabilities, Opus 4.8 was more successful at creating reproducible PoV and validated patches (patches that prevent the PoV from triggering after being applied). Opus 4.8 was nearly 4 times more successful end-to-end (38.9% vs 10.5%).

Comparison to AIxCC Results

To compare each model’s performance against the CRSs that competed in the AIxCC, we’ll look at the subset of challenge problems and injected vulnerabilities that were used in the final round of the AIxCC only: 63 injected vulnerabilities across 32 C and 16 Java challenge problems.

Before we get into the results, it’s worth noting that this isn’t a true apples-to-apples comparison, for several notable reasons: 

‍

1. There are clear disparities in resources, runtime, format of the challenge runs, and how rigorously PoVs/Patches were evaluated in the AIxCC versus this experiment. 
2. The experiments we ran did not benefit from pre-existing infrastructure or the competition API that was present in the AIxCC that helped CRSs avoid producing excess false positives. 
3. Although DARPA’s challenge problems were just made public last month, and after the training cutoff for both models, they were likely trained on LLM interactions that CRSs made when processing these challenges during the AIxCC.

‍

For these reasons, we excluded likely false positives produced by the models and 0-days discovered by CRSs in the actual competition in the results below. The comparison here should be treated as performance estimates, not head-to-head results.

Both general access models outperformed every AFC finalist in vulnerability detection. Overall, Opus 4.8 found 37 (58.7%) and GPT-5.5 found 46 (72.5%) of the injected vulnerabilities, both higher than the best AFC team (AT at 52.4%). Despite finding more vulnerabilities, both models struggled to produce reproducible PoVs and valid patches from these detections. Opus 4.8 was able to produce 29 PoVs and 23 patches and GPT-5.5 was able to produce 10 valid PoVs and patches, leading to a 68.2% and 21.7% overall accuracy for PoV/Patch submissions, respectively.

‍

Based on these results, Opus 4.8 performed approximately on par with the 3rd place winning team (RoboDuck), and GPT-5.5 would have placed second to last. It’s clear that advancements in general access AI models and their harnesses have made them more capable compared to CRSs built for the AIxCC. Still, purpose built systems reign for achieving high accuracy, a key factor for those looking to deploy these systems in the real-world.

Key Takeaways

The experimental results suggest that AI-based vulnerability detection has become commoditized. Dramatic improvements to models and harnesses over the last year have placed them on par with custom large-scale CRS pipelines with respect to vulnerability detection. Further, the costs for accomplishing this are now wildly cheaper than the AIxCC, even when accounting for subsidies on subscription plans. We spent ~$400 in model subscriptions and ~$200 in compute to run this experiment, and even without subscription plans the total API cost for this experiment would still be a fraction of the total allowable budget for an AIxCC competitor (~$3K versus $135K). 

‍

AI-based vulnerability detection is now cost-competitive with conventional fuzzing infrastructure.

However, finding vulnerabilities is not and has never been the hardest part of security analysis. The hardest parts are prioritization, producing reliable evidence, and quality patches for a vulnerability - these challenging problems remain in 2026.

Today’s general access models produce fewer reproducible PoVs, fewer validated patches, and substantially more false positives than last year’s models running in custom harnesses. 

‍

Custom harnesses are significantly more capable at producing reliable artifacts and that’s what matters in practice. The good news is that these two approaches aren’t in opposition - as baseline models improve their capabilities they create a rising tide that lifts all boats for custom and general access harnesses alike. In the future, the decision to use a commodity harness or build a custom one will continue to depend on the problem at hand and your tolerance for cost, false positives, and reliability. 

‍

[*1] NOTE: Five challenges were excluded from analysis due to various formating / technical issues.