How LLM coding agents are evaluated, trained, and harnessed — the 2025–2026 research record, distilled to its data. Built from 30+ primary sources; every claim checked against the paper that made it.
Part I · The Bench — slides 02–08 Part II · The Gym — slides 09–16 Part III · The Scaffold — slides 17–21
State of the art on SWE-bench (→ Verified), per the benchmark's own record — the curve that made and then unmade the field's favorite number
| Milestone | Date | Best |
|---|---|---|
| SWE-bench launches — 2,294 tasks, 12 Python repos, from ~90K PRs | 2023-10 | 1.96 |
| Oracle-retrieval analysis ceiling, same paper | 2023-10 | 4.80 |
| Leaderboard pre-Verified (full / Lite) | 2024-08 | 20 / 43 |
| Verified launches — 500 human-screened tasks | 2024-08 | 33.2 |
| SOTA, six months before retirement | ~2025-08 | 74.9 |
| SOTA at retirement announcement | 2026-02 | 80.9 |
Verified exists because the raw benchmark under-measured models: 93 Python-experienced developers screened 1,699 sampled tasks and discarded 68.3% as broken or underspecified. The cleaned set doubled GPT-4o's score (16% → 33.2%).
One construction recipe — issue + repo snapshot + fail-to-pass tests — replicated across sizes, modalities, languages, and economies
| Benchmark | Date | Tasks | Scope | What it added | Debut best |
|---|---|---|---|---|---|
| SWE-bench | 2023-10 | 2,294 | Python · 12 repos | The recipe: real issues, median 51 regression tests per task | 1.96 |
| SWE-bench Lite | 2024 | 300 | Python · 11 repos | Cheap subset; self-contained bug fixes (+23 dev tasks) | — |
| SWE-bench Verified | 2024-08 | 500 | Python | Human screening (93 devs × 1,699 tasks; 68.3% discarded) | 33.2 |
| SWE-bench Multimodal | 2024-10 | 517 test | JavaScript · 17 repos | Screenshots/diagrams in issues; visual front-end bugs | 12.0 |
| SWE-Lancer | 2025-02 | 1,488 | Expensify · Upwork | Real dollars: $1M in historical payouts; e2e Playwright tests by 100 paid engineers; manager tasks too | 26.2 IC |
| SWE-bench Multilingual | 2025-05 | 300 | 9 languages · 42 repos | C, C++, Go, Java, JS, TS, PHP, Ruby, Rust | 43.0 |
"The SWE-Bench Illusion" (Microsoft Research + Purdue): two diagnostics that need no agent, no repo, no tools
A 23-point inside/outside gap on file identification, and ~2× the verbatim reproduction rate, are exactly the fingerprints training-set leakage would leave — and they motivated every contamination-resistant design on slide 08.
Two independent audits of the grading harness itself: UTBoost (ACL 2025) and OpenAI's 138-problem deep review
| UTBoost finding | Lite | Verified |
|---|---|---|
| Instances with insufficient tests | 7.7% (23/300) | 5.2% (26/500) |
| PASS_TO_PASS / FAIL_TO_PASS annotation errors | 54.6% | 54.2% |
| "Passing" agent patches actually erroneous | 176 | 169 |
| Leaderboard entries affected | 40.9% | 24.4% |
| Ranking changes after re-grading | 18 | 11 |
The 26 Verified failures survived the 93-developer screen. Re-grading demoted Amazon Q Developer from sole 1st (55%) into a tie at 53.6%. Cost of the audit: ~$1.60 per instance (GPT-4o test generation + intramorphic oracle).
59.4% of the hard residue had material flaws in test design or problem description — "extremely difficult or impossible even for the most capable model or human to solve." The residual ceiling wasn't model failure; it was benchmark artifact.
The red-team that ended it: GPT-5 probing three frontier models over 15-turn elicitation
| Date | Event |
|---|---|
| 2025-06 | Illusion + UTBoost preprints land within a week of each other |
| 2025-09 | SWE-Bench Pro released as the "post-saturation" successor |
| 2026-02 | Verified retired; SOTA frozen at 80.9% |
| 2026-03 | django-11451 contamination independently replicated (5/5 byte-identical) |
| 2026-07 | Pro itself audited: ~30% of public tasks reported broken |
Best reported result at each benchmark's release — different models & harnesses; a narrative of difficulty, not a head-to-head
SWE-bench-Live re-ran the identical agent, model, and settings on both benchmarks:
A 2.2× gap with every variable held constant except task freshness — plus a within-Live split: 22.96% on SWE-bench-origin repos vs 18.89% on novel repos.
Three defenses — harder tasks, more languages, fresher data — and construction itself turned agentic
| Benchmark | Size | Coverage | Contamination defense | Frontier @ release | Construction |
|---|---|---|---|---|---|
| SWE-Bench Pro Scale AI · 2509.16941 | 1,865 = 731+858+276 | 41 repos; enterprise-grade, multi-file, hours–days tasks | GPL copyleft public/held-out; 276 tasks on never-public startup code | GPT-5 23.3 · Opus 4.1 22.7; commercial all <20 | Human-verified; SWE-Agent scaffold |
| Terminal-Bench 2.0 ICLR 2026 · 2601.11868 | 89 | Terminal work: SWE, ML, security, sysadmin, data science | Hand-built tasks; 89 kept of 229, 3 reviewers each | <65 (best 63: Codex CLI + GPT-5.2) | Unique env + human solution + test suite per task |
| SWE-bench-Live MSR · 2505.23419 | 1,319 → ~1,890 | 93 → 223 repos, Python | Only post-2024 issues; monthly refresh | 19.25 (Claude 3.7 + OpenHands) | RepoLaunch: LLM agent builds each Docker env end-to-end |
| SWE-rebench Nebius · 2505.20411 | 21,336 | 3,468 Python repos | Issue-date vs model-release tracking; flagged leaderboard | time-sliced monthly splits | Fully automated 4-stage funnel from ~450K PRs |
| Multi-SWE-bench ByteDance · 2504.02605 | 1,632 | Java, TS, JS, Go, Rust, C, C++ | — | 9 models × 3 scaffolds | 68 expert annotators filtered 2,456 candidates |
| SWE-PolyBench AWS · 2504.08703 | 2,110 | JS 1,017 · TS 729 · Py 199 · Java 165; bugs+features+refactors | — | Aider 14.1 · SWE-agent 10.2 · Agentless 7.8 | Adds CST node-level retrieval metrics beyond pass-rate |
The meta-move: evaluation infrastructure became agent-built (RepoLaunch environments, UTGenerator test audits, SWE-rebench's automated funnel) — and the same containerized fail-to-pass instance now serves as eval item and RL episode. Which is Part II's story.
The stage order below is the slide order that follows — substrate first, then each training stage, then integration and results
Executable repository environments with tests — every stage below consumes them. Curated, mined, or fully synthesized.
→ slide 10Repo-level code at scale; multi-hundred-language coverage; long-context packing.
K2: 15.5T tokens, MuonClip · QCN: 92→370 languages, 262K ctx — disclosed in the frontier reportsAgent-shaped data between code and trajectories: trajectory-native corpora, function-aware FIM.
daVinci-Dev: 58.5% SBV with no RL · FIM: +2.8…+5.4Rejection-sampled expert trajectories from frontier teachers in executable envs.
SWE-smith: 5,016 traj → 40.2% · SWE-Gym: 491 traj → +14 absMulti-turn agentic RL against test execution; GRPO-family; parallel container fleets.
DeepSWE: 512 containers/iter · Qwen: 20,000 parallel envsWhat frontier reports disclose when all stages run together — K2, Qwen3-Coder, Qwen3-Coder-Next.
→ 15Fourteen months of open-weights scores on one chart: 20.6 → 71.3.
→ 16Formal frame for the whole part: the 500+-work agentic-RL survey (TMLR) casts preference tuning as a degenerate single-step MDP and agentic training as a temporally extended POMDP with execution-verifiable rewards — "unit-test passes, symbolic verifier success." Its code-agent taxonomy: generation → iterative refinement → automated software engineering.
Three generations: curate real repos → synthesize bugs into real repos → synthesize everything
| Corpus | How built | Key training result |
|---|---|---|
| SWE-Gym ICML '25 | Real issues + envs + tests; ~200h manual setup; Raw split 64,689 (no envs) | 491 trajectories → +14 abs on Verified (32B) |
| SWE-smith NeurIPS '25 D&B | 5 bug-synthesis strategies: LM-modify 17,887 · procedural 15,641 · combine 10,092 · LM-rewrite 4,173 · PR-mirror 2,344 | PR-mirror bugs train best; LM-modify worst (5.7% vs 7.8–8.8%) |
| SWE-rebench | 4-stage automated funnel: ~450K PRs → 153.4K → 21,336 validated | Feeds Nebius RL (7,249 tasks, slide 13) + daVinci mid-training (slide 11) |
| SWE-Playground CMU | Projects, tasks, tests all generated from scratch; OpenHands sandbox | 2–3× denser trajectories (39K tok, 36 tool calls); new task types transfer: SWT-Bench 18.1% @32B |
Between static code and agent trajectories — two papers put numbers on the gap
| Result (SBV) | Score |
|---|---|
| daVinci-Dev-72B — mid-train + SFT, no RL | 58.5 |
| daVinci-Dev-32B | 56.1 |
| Kimi-Dev-72B under aligned agentic setting (their comparison) | 48.6 |
At less than half the comparison baseline's mid-training budget. Rollouts generated by GLM-4.6 driving SWE-Agent; mid-train-only checkpoints released for others to post-train.
Mask whole functions picked by dependency-graph + complexity×inferability scoring — betting that the agent's act-observe-continue loop is isomorphic to a call site. 2.6B tokens, 968 repos, inserted before existing post-training.
Repairs collateral damage too: agentic post-training alone costs a 14B model −13.1 LiveCodeBench and −7.4 BFCL; mid-training recovers the six-benchmark average by +3.5 while keeping the agentic gains. Behavior shifts: 3.3 → 7.4 edits per solved task, recovery rate 24.8 → 28.8%.
The workhorse recipe of 2025 — sample a frontier teacher inside an executable environment, rejection-filter by test execution, fine-tune an open model
| Pipeline | Teacher → student | Trajectories | Result (SWE-bench Verified) | What it established |
|---|---|---|---|---|
| SWE-Gym ICML '25 | GPT-4o + Claude 3.5 Sonnet → 32B | 491 (~19 turns, ~19K tok each) | +14 abs → 20.6 pass@1 | Sample efficiency; verifier best-of-16 → 32.0 (LoRA verifier beats full FT, 29.8 vs 27.2 best@8) |
| SWE-smith NeurIPS '25 D&B | Claude 3.7 Sonnet → Qwen2.5-Coder-32B | 5,016 | 40.2 (+33.4 over base) | Scale: open-weights SoTA at publication; 7B variant on 2.0K traj → 15.2 |
| R2E-Gym baseline | → 7B / 32B | 3,321 | 19.0 @7B | The curated-env comparison line in SWE-Playground's tables |
| SWE-Playground CMU | Gemini 2.5 Pro ensemble → Qwen2.5-Coder | 704 (2–3× denser: 39K tok, 36 calls) | 17.0 @7B · 31.2 @32B | Density over count; task diversity transfers (SWT-Bench 18.1 @32B, best reported) |
| Qwen3-Coder-Next frontier practice | Qwen3-Coder-480B → 80B-A3B | — (6 scaffolds, 21 tool templates) | 70.6–71.3 (after RL) | Distilling across six scaffolds as deliberate anti-overfitting — then RL on top |
The verified 2025 record spans the whole reward-design spectrum — and disagrees productively
| Recipe | Base → result (SBV) | Reward signal | Algorithm | Scale / infra | Distinctive evidence |
|---|---|---|---|---|---|
| SWE-RL Meta FAIR · NeurIPS '25 | Llama-3.3-70B → 41.0 | No execution: difflib similarity to gold patch; −1 if malformed | GRPO | 273K PR seed instances | Generalizes out-of-domain (math, MMLU up) where SFT on same data degrades |
| Agent-RLVR Scale AI | Qwen2.5-72B 9.4 → 22.4 (→27.8 + RM rerank) | Unit tests on 817 envs + teacher guidance on failures (plans, error feedback, file pointers) | Offline iterative DPO | 8,186 trajectories; 4–10 h/epoch on H100s | Guidance ablation 19.8→22.4; also 14B 6.8→18.0, 32B 8.8→21.6; beats SFT-same-data (20.8) |
| DeepSWE Agentica + Together | Qwen3-32B → 42.2 (59 with TTS) | Pure execution: sparse 0/1 test pass in 5-min limit; no SFT, no teacher | GRPO++ (DAPO clip-high, Dr.GRPO, leave-one-out; no KL) | 4.5K R2E-Gym tasks · 512 containers/iter · 64 H100 × 6 days | ~23 → 42.2 in 200 RL steps; everything open: weights, data, code, W&B logs |
| Nebius 72B 2508.03501 | Qwen2.5-72B 11 → 39 (RFT alone: 20) | Binary test success + length penalty; no teacher | Rejection FT → synchronous DAPO, 2 stages (65K→131K ctx, 40→80 turns) | 7,249 SWE-rebench tasks · 128 H200s | 35 / 31 on decontaminated SWE-rebench May/June splits — the contamination-aware readout |
Three escapes from sparse reward: densify it (similarity proxy), guide it (teacher hints + off-policy preference pairs), or out-scale it (parallel container fleets until on-policy sparse reward works). The SFT-vs-RL question now has two same-data answers pointing the same way — toward RL.
SWE-bench Verified pass@1, same model family before and after the recipe
What the three most detailed open model reports disclose about running the full pipeline
| Model | Params | Agentic training data | RL story | SBV | Rare disclosure |
|---|---|---|---|---|---|
| Kimi K2 Moonshot · Jul '25 · 2507.20534 | 1.04T / 32.6B act | Synthesis pipeline over 3,000+ real MCP tools + 20,000+ synthetic tools; 15.5T pre-train tokens | Joint RL with real + synthetic environments; MuonClip optimizer, zero loss spikes | 65.8 71.6 multi | Full optimizer recipe at 1T scale; base + instruct checkpoints released |
| Qwen3-Coder Alibaba · Jul '25 · blog | 480B / 35B act | "Hard to solve, easy to verify" task farming; 256K native ctx (1M YaRN) | Long-horizon RL on 20,000 parallel environments on Alibaba Cloud | 69.6† | Environment-fleet scale as the stated bottleneck |
| Qwen3-Coder-Next Alibaba · Feb '26 · 2603.00729 | 80B / 3B act | ~800K synthesized verifiable SE tasks (~9 langs, real PRs → Docker) + ~600B repo tokens; SFT distilled from 480B teacher across six scaffolds | Env feedback in mid-training and RL; MegaFlow K8s/Argo orchestration (rollout→eval→analysis) | 70.6–71.3 Pro 42.7 | "Reinforced Reward-Hacking Blocker": agents learned to fetch gold patches over the network; blocker bans repo-link + git/curl/wget combos; hacking intensifies with capability |
Every trained-model release in the verified record, Dec 2024 – Feb 2026 · line = running best open score
| Model | Date | SBV% |
|---|---|---|
| SWE-Gym ft. 32B | 12/24 | 20.6 |
| Llama3-SWE-RL-70B | 02/25 | 41.0 |
| SWE-agent-LM-32B | 04/25 | 40.2 |
| Agent-RLVR 72B | 06/25 | 22.4 |
| DeepSWE-32B | 07/25 | 42.2 |
| Devstral Small 1.1 | 07/25 | 53.6 |
| Qwen3-Coder-480B | 07/25 | 69.6 |
| Kimi K2 (1.04T) | 07/25 | 65.8 |
| Nebius 72B | 08/25 | 39.0 |
| GLM-4.6 | 09/25 | 68.0 |
| SWE-Compressor 32B | 12/25 | 57.6 |
| daVinci-Dev-72B | 01/26 | 58.5 |
| Qwen3-Coder-Next | 02/26 | 71.3 |
Scaffold ≡ harness: all the software around the model that turns next-token prediction into a coding agent — the field even renamed one term to the other mid-2026
Proposes the next action from the context it is shown: a shell command, a file edit, a plan, a patch.
Control loop (when to think, act, retry, stop) · prompt & context assembly (what the model sees) · tool dispatch & edit formats · context compaction · state & model routing.
Repo checkout, shell, test runner, Docker sandbox — returns observations and, in training, the reward.
| Taxonomy layer (source-code census of 13 open agents at pinned commits) | What varies | Census finding |
|---|---|---|
| 1 · Control architecture | Loop type; 5 composable primitives: ReAct, generate-test-repair, plan-execute, multi-attempt retry, tree search | 1 fixed pipeline (Agentless) · 7 ReAct (SWE-agent, OpenHands, Codex CLI, Gemini CLI, mini-swe-agent, Cline, OpenCode) · 2 phased (AutoCodeRover, Prometheus) · DFS (DARS) · MCTS (Moatless); 11 of 13 compose ≥2 primitives |
| 2 · Tool & environment interface | Tool count & categories, edit formats, execution isolation | Tools range 0 (Aider) → 37 (Moatless); converges where external constraints bind |
| 3 · Resource management | Context compaction, state management, multi-model routing | 7 distinct compaction strategies (truncation, sliding window, LLM summarization, selective dropping … none); diverges exactly on the open questions |
Part III in that structure: measure its effect → 18 watch it evolve → 19 engineer it (context, SDKs, meta-agents) → 20 or delete it → 21
The scaffold was worth ~2× in 2025; at the frontier the spread narrows — but engineering still buys points
| Evidence · model fixed | Spread |
|---|---|
| SWE-Gym zero-shot, Qwen2.5-Coder 7B/32B: MoatlessTools 7/19 vs OpenHands 1/3 (Lite) — scaffold decides whether a weak model functions at all | 7× |
| SWE-bench-Live: OpenHands 19.25 vs SWE-agent 17.13 (Claude 3.7) | 1.12× |
| SWE-PolyBench: Aider 14.1 vs Agentless 7.8 (Claude 3.5) | 1.8× |
| Qwen3-Coder-Next across 3 scaffolds: 70.6 / 71.1 / 71.3 | 1.01× |
| SWE-Bench Pro: CCA 59.0 vs OpenAI-reported 56.0 (GPT-5.2) | +3.0 pts |
| Terminal-Bench 2.0 authors: "model choice matters more than scaffold" (Codex CLI + GPT-5.2 63 vs Terminus 2 + Opus 4.5 58) | — |
The pattern: harness sensitivity is inversely proportional to model capability — enormous for weak models, ~1 point for frontier ones on familiar benchmarks — yet deliberate scaffold engineering (slide 20) still moves frontier scores on hard benchmarks by high single digits.
First controlled longitudinal study: model frozen (Qwen3-Next-80B-A3B via vLLM), 35 sequential Qwen Code CLI releases replayed — 3,500 task executions
| Component changed | Quality outcome across releases |
|---|---|
| LLM provider layer | Most frequently associated with degradation |
| Context management | Most frequently associated with degradation |
| Extensibility, security | Safe or neutral |
| Ecosystem velocity (RQ0) | Rate |
|---|---|
| OpenCode releases | 18.0 / week |
| OpenCode median release interval | 0.12 days |
| Gemini CLI issues in 224 days | 9,951 |
| OpenCode issues in 278 days | 8,621 |
| Codex commits (total studied window) | 9,521 |
Verification note: the paper's replication package (all 35 versions, scripts, results) is promised "upon acceptance" — not yet public. Queen's University; v2 retitled "scaffolding" → "agent harness" throughout.
Context management became trainable; scaffolds became SDKs with SRE-grade receipts
| System | Idea | Numbers |
|---|---|---|
| CAT / SWE-Compressor Beihang · ACL 2026 Findings · 2512.22087 | Compaction as a callable tool the agent invokes at milestones; workspace = stable semantics + long-term memory + short-term detail. Trained via 20K trajectories with injected context-management actions (avg 87.4 steps, 4.22 actions each) | 57.6% SBV (Qwen2.5-Coder-32B base, ≤500 rounds); folded history 15.6K → 4.7K tokens (~30% ratio); beats ReAct + static compression under bounded context |
| OpenHands Agent SDK MLSys 2026 + ACM badges · 2511.03690 | V0 monolith → 4 packages on event-sourced state; native sandboxing, lifecycle control, multi-LLM routing, security analysis (absent from OpenAI/Claude/Google SDK comparison table) | 15-day production rollout: system failures −61% (78.0 → 30.0 /1K convs; infra 69.8 → 0.0); event persist 0.20 ms median; SBV 76.6 (Opus 4.5); V0→V1 gain is model-conditional: Sonnet 4.5 +8.2, Sonnet 4 ±0 |
| Confucius Code Agent Meta + Harvard · 2512.10398 | A meta-agent that builds, tests, and refines coding agents (build-test-improve); hierarchical memory + persistent cross-session notes; AX/UX/DX design split | 59.0% Pro Resolve@1 w/ GPT-5.2 (vs 56.0 reported); 74.6% SBV w/ Claude 4 Sonnet; ablations: context mgmt +6.6 pts; notes 53.0 → 54.4% with turns 64→61 and tokens 104K→93K |
Triangulation: the longitudinal study says context-management changes are where harnesses break (slide 19); the taxonomy says compaction is where designs diverge most (slide 17); CAT shows training the model to manage its own context beats bolting heuristics onto the harness; OpenHands' default summarizing condenser cuts API cost up to 2× with no performance loss.
Verification notes applied: CCA's institution corrected to Meta + Harvard (not Meituan); its 54.3% v1 "SoTA" claim was withdrawn a day after posting and re-framed — the 59.0 headline arrived in v6. OpenHands' SBV 76.6 sits below the 79.2 published SoTA it cites — SoTA on 3 of 5 suites (GAIA, Commit0, SWE-bench Multimodal), not on Verified.
Bash as the only tool (no tool-calling API), strictly linear history, stateless subprocess.run execution. Marketed "some 100 lines of python" (190 on disk — we counted). 65% SBV with Claude Sonnet 4 at launch, 5 points shy of Anthropic's own scaffold; >74% today. Rationale: 2024 scaffolds compensated for chat-tuned models, but 2025 models are trained for agentic coding — so the scaffold can dissolve. Now the model-centric baseline for SWE-bench's "bash-only" leaderboard and for RL work that must not overfit a harness, the role Terminus 2 plays for Terminal-Bench. Against slide 20's SDKs, this is the live design argument of 2026.
Convergence thesis. Evaluation, training, and harness research now share one substrate — the containerized fail-to-pass repository. Benchmarks ship RL twins (Multi-SWE-bench→Multi-SWE-RL); models are trained inside the scaffolds they're scored in; harness skills are trained into weights. Scores are properties of systems, not models.
SWE-Gym Lite = 230 (not 234) · QCN Pro = 42.7 (not 56.2, column misread) · CCA = Meta+Harvard, 59.0 on GPT-5.2 · harness-study replication unreleased · OpenHands +8.2 is Sonnet-4.5-only · SWE-smith 7B comparison = 2.0K trajectories