VulcanBench Eval Suite 3: DeepSeek and Grok Tie at the Top

On August 1, 2026, Morgan Linton released Eval Suite 3 of VulcanBench, a benchmark built from 23 frontier-hard software-engineering tasks drawn from real merged open-source pull requests. Unlike leaderboard-style tests that reward broad general knowledge, VulcanBench focuses on what an autonomous coding agent actually does in production: read a repo, understand the problem, patch it, and pass CI.

This round pits DeepSeek, xAI, Anthropic, OpenAI, and Moonshot against one another across low, medium, and high reasoning-effort settings. The results are a wake-up call for anyone who still assumes a bigger model name automatically equals better agentic code performance.

What VulcanBench Suite 3 measures

VulcanBench uses Docker-sandboxed agent runs and reports pass@1 — the percentage of tasks solved on the first attempt. No retries, no majority voting, no human hand-holding. The 23 tasks come from merged OSS PRs, which means they are real bugs, real features, and real integration challenges, not sanitized academic exercises.

The benchmark also tracks cost per suite at list API prices, so each bar in the ranking is annotated with what it would actually cost to reproduce the run.

The headline: three models hit 91%

The top of the leaderboard is unusually tight. Three entries land at 91% pass@1:

• DeepSeek V4-Flash (medium effort) — $2.04 per suite
• Grok 4.5 (medium effort) — $6.67 per suite
• Grok 4.5 (high effort) — $13.16 per suite

DeepSeek V4-Flash does it for less than a third of Grok's cheapest 91% run. That is the story of the chart: DeepSeek's post-trained V4-Flash checkpoint matches the best frontier agents while staying dramatically cheaper.

Right behind them is Claude Fable 5 (low effort) at 89%, then a cluster at 87%:

• DeepSeek V4-Flash (low) — $0.95
• DeepSeek V4-Flash (high) — $18.02
• GPT-5.6 Sol (high) — $15.00

The full rankings

Two patterns jump out. First, Grok 4.5 is flat across medium and high effort — it hits 91% at both settings, so paying more does not buy better results on this suite. Second, DeepSeek V4-Flash actually loses performance at high effort, dropping from 91% at medium to 87% at high. More reasoning tokens do not always translate to better engineering decisions.

Effort curves tell a different story

The lower half of the image plots how each top model responds to reasoning effort:

• DeepSeek V4-Flash peaks at medium effort (91%) and falls back to 87% at low and high. The medium setting is clearly the sweet spot.
• Grok 4.5 ramps from 83% at low to 91% at medium, then stays flat through high.
• Claude Fable 5 dips at medium effort (81%) before recovering to 85% at high, which is unusual — low effort is its best setting here.
• GPT-5.6 Sol improves monotonically with effort, going from 78% at low to 83% at medium and 87% at high.

These curves matter for product teams. If you are building a high-volume coding agent, the right model is not necessarily the one with the highest peak score. It is the one that hits an acceptable pass rate at the lowest cost per task.

Cost efficiency: the unsung ranking

At $0.95 for 87% pass@1, DeepSeek V4-Flash at low effort is the cheapest way to get near-frontier performance. At $2.04 for 91%, it is the cheapest way to hit the top tier. No other model comes close on either metric.

By contrast, Claude Fable 5 at low effort delivers 89% but costs $23.16 — more than ten times the DeepSeek low-effort run. Grok 4.5 at high effort also hits 91%, but at $13.16 it is over six times the price of DeepSeek's 91% run.

For teams running agents on hundreds or thousands of tasks per day, that gap decides whether an agentic feature is profitable or experimental.

What about Kimi and Claude Haiku?

Kimi K3 at extra-high effort lands at 74%, the lowest full-suite score in this release. The footnote notes it has partial coverage on 19 of 23 tasks, so the final number may shift once full coverage is available. Still, it is a surprising result for a model that has performed strongly on other coding leaderboards.

Claude Haiku 4.5 at default effort scores 76% with partial coverage on 21 of 23 tasks. That puts the smaller Anthropic model behind every full-configuration entry from DeepSeek, Grok, and GPT-5.6 Sol, though it is also positioned as a faster, cheaper variant.

Caveats to keep in mind

Every benchmark has blind spots, and VulcanBench Suite 3 is no exception:

1. Partial coverage. Claude Fable 5, Kimi K3, and Claude Haiku 4.5 ran fewer than 23 tasks because of refusals or safety filters. Their scores are not perfectly comparable to full-suite runs.
2. Single snapshot. The results are from one evaluation run on August 1, 2026. Model behavior changes with checkpoint updates and post-training.
3. Narrow domain. The 23 tasks are real OSS PRs, but they are still a small sample. A model that does well here may struggle on proprietary codebases with private frameworks.
4. Cost is list price. Actual spend depends on caching, batching, retries, and any negotiated discounts.

Bottom line

VulcanBench Suite 3 confirms what many developers have started to suspect in mid-2026: reasoning-effort tuning and post-training matter at least as much as base model size. DeepSeek V4-Flash tops the chart at a fraction of the cost, Grok 4.5 is competitive but plateaus early, and the big-name API models need higher effort settings to keep up.

If you are choosing a coding agent today, the benchmark suggests a simple heuristic: start with DeepSeek V4-Flash at medium effort, validate on your own repo, and only escalate to Grok 4.5 or GPT-5.6 Sol at high effort if your specific tasks show a measurable gain.

For now, the leaderboard belongs to the model that delivers 91% for $2.

---

Source: VulcanBench Eval Suite 3 — Model Rankings, published August 1, 2026. Image provided by the benchmark author.