Gemini 4 Carbon's Cyber Lead: Real in Direction, Unproven in Size

Gemini 4 Carbon's cybersecurity lead is probably real in direction and unproven in size. Google computed two of its three headline security scores itself and took the rival figures from the rivals' own reports,⁠¹ and the one score an outside party ran shows a three-point lead on 120 tasks.⁠²

Google's claim: state of the art across the defense lifecycle

Reported Google's 28 October announcement says Carbon "achieves state-of-the-art performance across the cybersecurity defense lifecycle," which it defines as vulnerability detection, validation and remediation: finding a flaw, confirming it is real, and patching it.⁠³ It backs that with three external benchmarks, meaning fixed task sets scored the same way for every model: 88.2% on SEC-bench Pro for finding flaws, 54.8% on ExploitGym for exploitation and 71.0% on CWE-bench for patching.⁠³ VentureBeat repeated the framing, reporting that Google claims the top spot on most common enterprise benchmarks and that Carbon posts the highest result in 15 of the 17 rows of Google's own table.⁠⁴

Carbon has no public API, the programmatic interface developers call to use a model, so a reader cannot rerun any of these numbers.⁠⁴

A benchmark score also measures a model on a fixed task set, which is a separate question from how AI-assisted vulnerability discovery shows up in the monthly patch counts Microsoft publishes. This article deals only with the first.

The strongest case for Carbon's lead

Google's table lists Carbon first on all three security benchmarks. On ExploitGym it shows 54.8% against 42.4% for GPT-6 Astra and 34.5% for Claude Opus 5.5, and on SEC-bench Pro 88.2% against 85.4% for Astra and 83.4% for Gemini 4 Argon.⁠⁵ Carbon also ranks first on Artificial Analysis's Intelligence Index v5.0 at 59, one point above Opus 5.5, in a ranking run by an outside evaluator that combines many tests into one score.⁠⁴

The best evidence is the test Google did not run. CWE-bench is an independent benchmark, run by a party other than the model's maker. It uses a 120-task held-out set from Collinear AI, meaning tasks withheld from public release so models are less likely to have seen them. Carbon ranks first at 71% programmatic pass@1, the share of tasks solved on a first attempt and checked by automated tests rather than a judge, and its judge-panel pass@1 of 71% compares with 67% for Opus 5.5 and 66% for Astra.⁠² Google's methodology document also states which scores it computed itself, which makes the sorting below possible.⁠¹

On direction, the steelman wins, and the position here is built around that.

Two of the three scores are Google's own runs

SEC-bench Pro was run by Google on a 183-task May 2026 JavaScript engine split, with pass@1 averaged over three attempts in Google's Antigravity harness, the scaffold of tools and prompts a model runs inside. Rival figures came from their system cards and technical reports.⁠¹ The table Data Studios reproduced has no SEC-bench Pro entry for Opus 5.5 or Fable 5.1, and Astra's 85.4% sits 2.8 percentage points below Carbon.⁠⁵ By GetCoreTech's arithmetic, 2.8 points is about five tasks out of 183. The binomial standard error, the random wobble expected in a pass rate from scoring a finite number of tasks, is about 2.4 points for a score near 88% on 183 tasks.

ExploitGym used one attempt per task, no internet and the Antigravity harness, in the default setting with mitigations off, meaning exploit defenses were switched off. A task counts only if the exploit captures the target flag and an LLM judge confirms the intended vulnerability was used.⁠¹ Google measured a 6-hour and a 10-hour timeout, the time limit per task, and the announcement gives one number. The results table in the evaluation PDF did not come through as text, so GetCoreTech could not confirm which timeout the 54.8% reflects. Nothing in the document says the rival scores used the same timeout, harness or mitigation setting.⁠¹

The biggest margin in the package, 12.4 points over Astra, sits on the benchmark where Google ran its own model and took the rivals' numbers from their reports.⁠¹⁠⁵

The outside test shows a three-point lead

CWE-bench is the cleanest of the three. Collinear AI ran 120 held-out tasks with four rollouts per model, each rollout a full attempt at the task set, at high reasoning, each model in its own vendor's harness: Antigravity for Carbon and Argon, Codex for Astra, Claude Code for Opus.⁠² Measured Carbon scored 71% programmatic pass@1, Argon, Astra and Grok 4.7 scored 68%, and Opus 5.5 scored 67%.⁠² Three points is roughly four tasks out of 120. At pass@4, where a task counts if any of four attempts succeeds, Carbon's 81% ties Grok 4.7.⁠²

Google says the external security benchmarks were run without guardrails, the safety filters built into the production product, so defenders using the guarded product may not see the same numbers.⁠³⁠⁴ On Artificial Analysis's factual-knowledge test, Carbon's accuracy rose while its hallucination rate, how often a model states something false, went from 4% to 11%.⁠⁴ That test covers factual questions and says nothing direct about vulnerability triage.

The objection: every score points the same way

The strongest objection is that three tests favoring Carbon add up to evidence. They do, but the tests are not independent in the way that matters. Two share Google's harness, Google's runs and Google's choice of rival numbers, so a bias in one carries into the other.

The cleaner comparison is Carbon against Argon, since both ran in Antigravity. Carbon leads by 4.8 points on SEC-bench Pro, 12.8 on ExploitGym and 3.0 on CWE-bench.⁠⁵⁠²

Opinion Carbon is probably the strongest model on these tests, and the evidence supports a lead over Argon far better than a lead over its rivals.

What would change this

Three results would change this conclusion. An outside group could run SEC-bench Pro and ExploitGym on Carbon and its rivals with one harness, one attempt count, one timeout and one mitigation setting. If Carbon's SEC-bench Pro lead then exceeds about five points, twice the sampling error computed above, the "unproven in size" half of the position falls. A refreshed CWE-bench held-out set, which Collinear says it will rotate on a schedule, could show Carbon ahead of the 68% group again.⁠²

Google could also publish Carbon's scores with its production guardrails on, the configuration defenders actually receive.⁠⁴ The Chrome team's tenfold patch-quality claim would need a named rival model and underlying results, which Google has not released.⁠⁴

What to do until Carbon is open

Use the announced scores to build a shortlist and keep them out of purchasing decisions. The generally available model is Argon, which Google's Gemini API pricing page still listed as the latest on 29 October.⁠⁵ Carbon's announced introductory price matches Argon's at $2 per million input tokens, the text sent to the model, and $10 per million output tokens, the text it writes back, rising to $4 and $20, and Google gave no end date for Carbon's introductory period.⁠³ Argon's introductory pricing ends 30 November.⁠⁴

Token-based pricing charges per unit of text processed, so identical token prices say nothing about the cost of a fixed vulnerability. Google has published no token counts, retries or run times for Carbon,⁠⁴ and the CWE-bench leaderboard shows average cost per rollout ranging from $0.79 for Opus 5.5 to $6.63 for Argon, with no figure yet for Carbon.⁠² When access arrives, test on vulnerabilities already fixed in your own repositories. Record cost per verified fix and the number of false reports, and compare against Argon in the same harness.