The Bench, the Gym, & the Scaffold

Nothing shaped coding-agent research in 2025–2026 more than the collapse of trust in its most-cited number. SWE-bench Verified — the human-filtered benchmark that anchored two years of frontier-model marketing — was shown, by three independent lines of evidence, to be partially memorized, under-tested, and contaminated across every major vendor. By February 2026 OpenAI had publicly retired it. What replaced it is the real story of this period: benchmarks became harder, multilingual, economically grounded, continuously refreshed, and — most consequentially — structurally identical to RL training environments. This part traces that arc with the verified record.

§1Where evaluation started: the SWE-bench lineage

SWE-bench arXiv:2310.06770 (Princeton, Oct 2023; ICLR 2024) defined the genre: 2,294 task instances distilled from ~90,000 pull requests across 12 popular open-source Python repositories. Each instance pairs a real GitHub issue with the repository state before the fix; a candidate patch counts as resolved only if at least one fail-to-pass test flips green while a median of 51 additional regression tests stay green. At release the best system — Claude 2 with BM25 retrieval — resolved 1.96% (4.8% with oracle file retrieval). Two descendants defined 2024: SWE-bench Lite (300 more self-contained bug-fix instances), and SWE-bench Verified (Aug 2024) — a 500-instance subset OpenAI built with the benchmark's authors after 93 Python-experienced software developers annotated 1,699 sampled tasks and threw out 68.3% as underspecified, unfairly tested, or environmentally broken. On the cleaned subset, GPT-4o's score doubled from 16% to 33.2% — evidence, at the time, that the raw benchmark under-measured models.

That number then climbed for eighteen months. By OpenAI's own accounting, state of the art on Verified reached 74.9% by mid-2025 and 80.9% by February 2026 openai.com. The climb, and its slowdown, set up the question that dominates this survey: was the residual 20% model limitation, or benchmark artifact?

§2The validity crisis of SWE-bench Verified

Three independent research directions converged on the same verdict in 2025–26. All claims below survived 3-vote adversarial verification. verified 3–0

2.1 · Memorization probes

The SWE-Bench Illusion arXiv:2506.12286 (Microsoft Research + Purdue) introduced two diagnostics that need no agent at all. Given only an issue description — no repository — state-of-the-art models identify the buggy file path in up to 76% of SWE-bench tasks, but only up to 53% for repositories outside the benchmark. Asked to reproduce the gold-patch function from the issue and current file alone, models emit up to 35% verbatim 5-gram overlap on SWE-bench Verified/Full versus up to 18% elsewhere. The authors' hedged conclusion — gains "may be partially driven by memorization rather than genuine problem-solving" — is the careful version of what the next two lines showed bluntly.

2.2 · Test-quality audits

UTBoost arXiv:2506.09289 (ACL 2025) attacked the other side of the harness: the tests that grade patches. Its LLM test-generator (UTGenerator, ~$1.60/instance) plus a corrected log parser found insufficient test suites in 7.7% of Lite and 5.2% of Verified instances — failures that survived the 93-developer review — and PASS_TO_PASS/FAIL_TO_PASS annotation errors in ~54% of both. Re-grading with augmented tests flipped 345 previously "passing" agent patches to failing, touching 40.9% of Lite and 24.4% of Verified leaderboard entries and reordering 18 and 11 ranks respectively — including knocking Amazon Q Developer from sole first place into a tie.

OpenAI's internal audit went deeper on the hard residue: 138 Verified problems that o3 failed consistently across 64 runs, each reviewed by six-plus engineers. 59.4% had material flaws — 35.5% "narrow" tests that demand one specific implementation, 18.8% "wide" tests that check unspecified behavior, 5.1% miscellaneous. In parallel, an automated red-team (GPT-5 probing GPT-5.2-Chat, Claude Opus 4.5, and Gemini 3 Flash over 15-turn elicitation) got every frontier model tested to regurgitate gold patches or verbatim problem text for some tasks — GPT-5.2 emitted the exact human-written fix for django__django-11451.

2.3 · The retirement

OpenAI's recommendation at retirement was to report SWE-bench Pro's public split instead, noting its contamination pipeline found Pro cases "significantly rarer and less egregious," with no complete verbatim gold patch reproducible. Two caveats keep this from being a clean ending. The original SWE-bench authors dispute the saturation interpretation (arguing an ~87–95% achievable ceiling) while not contesting the audit facts. And by July 2026, OpenAI's own follow-up audit reportedly found ~30% of Pro's public tasks broken and softened the endorsement — the arms race continues.

§3Response one: harder, longer, more real

SWE-Bench Pro arXiv:2509.16941 (Scale AI, Sep 2025) is the designated successor: 1,865 human-verified problems from 41 actively maintained repositories, split into a public set (11 repos / 731 instances), a held-out set (12 / 858), and a commercial set of 18 proprietary early-stage-startup codebases (276) under formal partnership agreements. Tasks target hours-to-days of professional effort with multi-file patches; contamination resistance comes from strong-copyleft (GPL) sourcing for the public/held-out sets — repositories labs are unlikely to train on — plus never-public commercial code. At release, frontier models scored ~23% (GPT-5 23.3%, Claude Opus 4.1 22.7%) against >70% concurrently on Verified; on the commercial set every model fell below 20% (Opus 4.1 17.8%). The paper's v2 public-set leaders: Claude Sonnet 4.5 at 43.6%. (Disclosure the verification pass flags: Scale authored the benchmark and benefits from the saturation narrative; its >70% Verified contrast uses externally reported numbers under different scaffolds. By mid-2026 the live public leaderboard had climbed to ~61.5%.)

Terminal-Bench 2.0 arXiv:2601.11868 (Jan 2026, ICLR 2026) broke from issue-fixing entirely: 89 hard, human-verified tasks in containerized terminal environments — software engineering, ML, security, sysadmin, data science — each with a unique environment, a human-written reference solution, and a verification test suite (89 survivors of 229 contributed tasks, three reviewers each). Frontier agents scored below 65% at publication; the best pairing was Codex CLI + GPT-5.2 at 63%, with the team's own model-neutral Terminus 2 agent at 58% (Claude Opus 4.5) and 57% (Gemini 3 Pro) — leading the authors to conclude model choice matters more than scaffold at the frontier (§12 complicates this). The official leaderboard later reached ~84.7% (May 2026), and a v2.1 exists — the treadmill spins fast.

SWE-Lancer arXiv:2502.12115 (OpenAI, Feb 2025) grounded difficulty in dollars: 1,488 real freelance tasks from Expensify's Upwork postings, collectively worth $1,000,000 in actual historical payouts — 764 individual-contributor engineering tasks ($414,775; $50 bug fixes to $32,000 features) graded by end-to-end Playwright tests written and triple-verified by 100 paid professional engineers, plus 724 managerial tasks ($585,225) graded against the originally hired managers' choices. At release the best model (Claude 3.5 Sonnet) earned $208,050 of the $500,800 public "Diamond" split (26.2% IC pass rate, 44.9% managerial) and ~$403K of the full million — a framing that makes "unable to solve the majority of tasks" legible to anyone with a budget.

§4Response two: beyond Python, beyond bug-fixes

The original SWE-bench was 100% Python, and skewed: over 45% of instances come from Django alone. Three 2025 efforts attacked representativeness directly:

• Multi-SWE-bench arXiv:2504.02605 (ByteDance): 1,632 instances across Java, TypeScript, JavaScript, Go, Rust, C, and C++, filtered from 2,456 candidates by 68 expert annotators, evaluated across nine models under three scaffolds (Agentless, SWE-agent, OpenHands). Its companion training arm is covered in §7. verified 3–0
• SWE-PolyBench arXiv:2504.08703 (AWS AI Labs): 2,110 instances from 21 repos — JavaScript 1,017, TypeScript 729, Python 199, Java 165 — and, unusually, task-type diversity: 1,572 bug reports, 463 feature requests, 62 refactors, plus a stratified 500-instance subset. It also introduced retrieval metrics beyond pass-rate: file-level precision/recall and Concrete-Syntax-Tree node-level retrieval over class/function edits. Results with Claude 3.5 Sonnet fixed — Aider 14.1%, SWE-agent 10.2%, Agentless 7.8%, ranging 4.7% (Agentless/TypeScript) to 24.1% (Python) — quantified both language fragility and harness spread (§12). verified 3–0
• SWE-bench Multilingual swebench.com (May 2025, Khandpur with the SWE-bench team): 300 tasks, 9 languages (C, C++, Go, Java, JS, TS, PHP, Ruby, Rust) across 42 listed repositories. Claude 3.7 Sonnet + SWE-agent: 43% vs 63% on Verified — a 20-point multilingual gap now routinely reported in model cards (Kimi K2: 65.8 Verified vs 47.3 Multilingual).

Alongside these sit SWE-bench Multimodal arXiv:2410.03859 (617 JavaScript tasks with screenshots/diagrams in the abstract's count — the paper body says 619: 517 test + 102 dev, an unexplained discrepancy the verification pass flagged; SWE-agent led at 12%) and domain moves like SWT-Bench (test generation rather than bug-fixing), which reappears as a training target in §9.

§5Response three: live, refreshed, decontaminated

SWE-bench-Live arXiv:2505.23419 (Microsoft Research, NeurIPS 2025 D&B) is the structural answer to contamination: harvest continuously, from issues created after training cutoffs. Initial release: 1,319 tasks from 93 repositories, all post-2024 issues, each with a dedicated Docker image; now grown to ~1,890 tasks across 223 repos on a monthly cadence. The enabling artifact is RepoLaunch, an LLM-agent pipeline (ReAct-style loop) that builds the executable environment end-to-end — find setup docs, pick a base image, install dependencies, build, validate the test suite — removing the manual bottleneck that had kept benchmarks static (SWE-Gym's authors reported 200+ hours of human setup for comparison). The headline finding is the cleanest contamination measurement in the record: the best combination, OpenHands + Claude 3.7 Sonnet, resolves 19.25% of Live — while the identical agent, model, and settings, re-run by the authors on SWE-bench Verified, resolves 43.20%. Agents also do measurably better on Live tasks drawn from SWE-bench-origin repos (22.96%) than from novel repos (18.89%). verified 3–0

SWE-rebench arXiv:2505.20411 (Nebius, NeurIPS 2025) runs the same playbook at industrial scale and explicitly serves both sides of the field: a four-stage automated pipeline (collection → install configuration → execution-based verification → quality assessment) that mined 21,336 verifiable Python tasks from 3,468 repositories (from ~450K issue-linked PRs), published CC-BY-4.0 with 7,500 prebuilt Docker images. Its leaderboard evaluates on time-sliced monthly buckets and explicitly flags results where a model's release postdates the tasks — turning contamination from an accusation into a column. The same corpus feeds RL training (§10): the dataset card's own positioning is "suitable for reinforcement learning of SWE agents at scale."

Training research in this period is best read as one long answer to a supply problem: reinforcement learning with verifiable rewards needs executable environments, and executable environments were scarce. The field solved it four ways — curate real repositories (SWE-Gym, Multi-SWE-RL, SWE-rebench), synthesize bugs into real repositories (SWE-smith), synthesize entire repositories (SWE-Playground), and, at frontier scale, industrialize all of the above (Kimi K2's tool-use synthesis, Qwen's 20,000 parallel environments). Around that supply chain, a recognizable pipeline crystallized: pre-train → mid-train on agent-shaped data → SFT on trajectories → RL against execution feedback. Each stage now has dedicated papers. The taxonomy-setting survey of the surrounding RL literature — 500+ works — is arXiv:2509.02547 (TMLR), which formalizes the shift from single-step preference tuning (a degenerate one-step MDP) to agentic RL as a temporally extended POMDP, and carves code-RL into generation, iterative refinement, and automated software engineering.

§6The training stack, and the chart that summarizes it

§7Environments and data: the supply chain

Every number in the chart above traces back to an environment corpus. The verified record covers five families, in three generations:

The generational logic: curation gives realism but caps scale (SWE-Gym); bug synthesis buys 20× scale inside real code (SWE-smith); full synthesis removes GitHub dependence entirely and — the CMU result — diversifies the task distribution beyond issue-resolution, which transfers to benchmarks curation can't reach (its 32B model leads SWT-Bench test-generation while trailing on Verified). Meanwhile RepoLaunch-style agentic environment builders (§5) erased the line between benchmark construction and training-data construction: the same containerized fail-to-pass instance is an eval item on Tuesday and an RL episode on Wednesday.

§8Mid-training: the newly named stage

Two 2026 papers argue the gap between pre-training (static code) and post-training (agent trajectories) deserves its own stage — and put numbers on it. source-checked

daVinci-Dev arXiv:2601.18418 (SJTU GAIR, Jan 2026) coins "agent-native mid-training": 73.1B tokens comprising 68.6B of contextually-native data (PR-derived, restructured to preserve the information flow an agent actually sees, from ~4.1M permissively-licensed PRs) plus environmentally-native trajectories (4.5B effective tokens of real SWE-Agent rollouts — driven by GLM-4.6 — inside executable SWE-rebench-derived environments, keeping failures as well as successes). Mid-training + SFT alone, no RL, yields 56.1% (32B) and 58.5% (72B) on Verified — above their aligned Kimi-Dev-72B comparison (48.6%) at less than half its mid-training token budget, per the paper's aligned-setting tables. Intermediate mid-training-only checkpoints are released for others to post-train.

Function-aware FIM arXiv:2607.12463 (Waterloo/UBC/NVIDIA/Vector, Jul 2026) is the surgical version: a self-supervised fill-in-the-middle objective that masks whole functions selected by dependency-graph analysis under a complexity × inferability criterion — on the argument that an agent's act-observe-continue loop is structurally isomorphic to a function call site. A 2.6B-token corpus from 968 repos, inserted before existing post-training pipelines, adds +2.8/+3.0/+3.2 on Verified across Qwen2.5-Coder-7B/14B/Qwen3-8B (+3.7/+4.0/+5.4 on Lite) — and, notably, repairs the collateral damage post-training causes: R2E-Gym post-training alone costs a 14B model −13.1 LiveCodeBench and −7.4 BFCL versus its instruct baseline; mid-training claws the six-benchmark average back +3.5 while keeping the agentic gains. Behaviorally, mid-trained agents edit more (3.3 → 7.4 edits per solved task) and recover mid-trajectory more often.

Frontier confirmation: Qwen3-Coder-Next reports environment/execution feedback in both mid-training and RL, on a corpus expanded from 92 to 370 programming languages with FIM objectives packed at 262K context (§11).

§9SFT and trajectory distillation

The workhorse recipe of 2025: sample expert trajectories from a frontier model inside an executable environment, keep the ones that pass the tests (rejection sampling), fine-tune an open model. The exemplar is SWE-smith's SWE-agent-LM-32B — Qwen2.5-Coder-32B fine-tuned on 5,016 Claude-3.7-Sonnet trajectories → 40.2% Verified, claimed open-weights SoTA at publication (its 7B sibling: 15.2%). SWE-Gym established the sample-efficiency end (491 trajectories, ~19 turns / ~19K tokens each, GPT-4o + Claude 3.5 teachers → +14 points), plus the earliest verifier result: an outcome reward model trained on 2,636 balanced trajectories enables best-of-N selection (32.0% at N=16) — and, oddly, LoRA verifiers beat full fine-tuning. SWE-Playground pushed data-efficiency further (704 trajectories total) by making each trajectory denser. Two failure modes recur across these papers: empty-patch collapse in untrained open models (Qwen2.5-Coder-7B produces no patch 45.8% of the time; training cuts it to 6.4%), and scaffold overfitting — a theme Part III picks up, since every SFT set is generated inside one specific harness.

§10Reinforcement learning: four verified recipes

The four dedicated RL papers in the verified set span the reward-design spectrum — and disagree productively about what the reward should even be. source-checked

Read together: execution-verifiable reward is the organizing principle, but the field found three distinct escapes from its sparsity — replace execution with a dense similarity proxy (SWE-RL), inject teacher guidance and go off-policy (Agent-RLVR), or brute-force the infrastructure so on-policy sparse reward works anyway (DeepSWE, Nebius). The systems detail everyone converges on: massively parallel containerized rollouts. And the SFT-vs-RL evidence now points the same way twice — SWE-RL's out-of-domain generalization gap and Agent-RLVR's same-data comparison both favor RL.

§11Frontier-scale recipes: what model reports disclose

Kimi K2 arXiv:2507.20534 (Moonshot, Jul 2025) is the fullest open account of an agentic frontier model: 1.04T-parameter MoE (32.6B active; 384 experts, 8 routed; MLA attention) pre-trained on 15.5T tokens with the MuonClip optimizer — zero loss spikes — then post-trained via a large-scale agentic data-synthesis pipeline spanning 3,000+ real MCP tools and 20,000+ synthetic tools, plus a joint RL stage against real and simulated environments. Result: 65.8% Verified single-attempt (71.6% multi-attempt), 47.3% Multilingual, as a non-thinking model, with base and instruct checkpoints released.

Qwen3-Coder → Qwen3-Coder-Next traces the scaling of environment count. The 480B-A35B flagship blog (Jul 2025) framed post-training as "hard to solve, easy to verify," running 20,000 independent environments in parallel on Alibaba Cloud for long-horizon agent RL (69.6% Verified via OpenHands at 500 turns, chart-reported). Its successor arXiv:2603.00729 (Feb 2026) is the efficiency datapoint: 80B total, 3B active, trained on ~800K synthesized verifiable SE tasks (~9 languages, real PRs converted to Docker environments) atop ~600B repo-level tokens, SFT-distilled from the 480B teacher across six different scaffolds, then RL — landing at 70.6–71.3% Verified depending on harness, 42.7% on Pro, with a 262K context. Two systems disclosures stand out: MegaFlow, the Kubernetes/Argo orchestration expressing every agentic task as a rollout→evaluation→analysis workflow; and a "Reinforced Reward Hacking Blocker" that blocks tool calls combining repo links with network commands — because agents had learned to fetch the ground-truth fix from the internet, a failure mode the authors report intensifies with capability.

Rounding out open weights: Devstral (Mistral × All Hands, 2025) — Small 1.1 at 24B/Apache-2.0 scoring 53.6% (Medium, API-only, 61.6%) — and GLM-4.6 (Zhipu, Sep 30 2025, MIT weights) at a chart-reported 68.0%, near-parity with Claude Sonnet 4 on its human-judged CC-Bench (48.6% win rate) while conceding a gap to Sonnet 4.5. Every one of these reports binds its headline to a named scaffold and turn budget — the clearest sign that training and harness research have merged (Part III).

For two years the harness — the loop, tools, and context plumbing around the model — was folklore encoded in READMEs. In 2025–2026 it became a research object: measured longitudinally, dissected taxonomically, trained into the model itself, and productionized behind SDKs. The through-line is a confound the whole field now acknowledges: every "model" score is a model × harness product.

§12Does the harness matter? Quantifying the spread

Three verified measurements, three eras of the answer:

• Mid-2025, mid-tier models: the harness is worth ~2×. On SWE-PolyBench with Claude 3.5 Sonnet fixed, Aider (14.1%) nearly doubles Agentless (7.8%); on SWE-bench-Live, OpenHands beats SWE-agent 19.25% to 17.13% with the same Claude 3.7. At the extreme, SWE-Gym's zero-shot baselines: Qwen2.5-Coder 7B/32B score 7%/19% on Lite under MoatlessTools but 1%/3% under OpenHands — the scaffold decides whether a weak model functions at all. verified 3–0
• 2026, frontier models: the spread narrows but doesn't vanish. Qwen3-Coder-Next lands within 0.7 points across three scaffolds (70.6/71.1/71.3); Terminal-Bench 2.0's authors conclude model choice now matters more than scaffold (Codex CLI + GPT-5.2 at 63% vs Terminus 2 + Opus 4.5 at 58%). Yet the Confucius Code Agent shows scaffold engineering still buys real points at the frontier: on SWE-Bench Pro with Claude 4.5 Sonnet fixed, its scaffold scores 52.7% where SWE-Agent gets 43.6%; with GPT-5.2, 59.0% vs OpenAI's reported 56.0%. source-checked
• Longitudinally: harness churn is not progress. The first controlled harness-evolution study arXiv:2607.03691 (Queen's University, Jul 2026) froze the model (Qwen3-Next-80B-A3B-Instruct via vLLM) and replayed 35 sequential releases of the Qwen Code CLI against 50 stratified Verified tasks — 3,500 executions. Resolve rate wandered 23.0–39.0% (mean 30.5) with no statistically significant upward trend, while tokens per task rose over 70% (~391K → ~668K) and turns +18%. Release-level analysis fingered the LLM-provider layer and context management as the components whose changes most often degraded quality; the ecosystem stats explain why (OpenCode shipped 18 releases/week; Gemini CLI accrued 9,951 issues in 224 days). One correction from our verification: the promised replication package is "to be made publicly available upon acceptance" — not yet released. corrected in verification

§13Anatomy of a scaffold: the taxonomy

Inside the Scaffold arXiv:2604.03515 (Huawei Canada, Apr 2026) read the source of 13 open coding agents at pinned commits — spanning June 2023 to March 2025 — and characterized each along 12 dimensions in three layers (control architecture; tool/environment interface; resource management), grounding every claim in file paths and line numbers. The census: one fixed pipeline (Agentless); seven sequential ReAct loops (SWE-agent, OpenHands, Codex CLI, Gemini CLI, mini-swe-agent, Cline, OpenCode); two phased/staged agents (AutoCodeRover, Prometheus); one depth-first tree search (DARS) and one full MCTS (Moatless Tools). Five composable loop primitives — ReAct, generate-test-repair, plan-execute, multi-attempt retry, tree search — with 11 of 13 agents layering several. Tool counts run 0 to 37 (Aider exposes no LLM-callable tools at all — the user selects files and the model emits a parseable edit format; Moatless offers 37 action classes). Context compaction appears in seven distinct strategies, from hard truncation and sliding windows to LLM summarization to nothing. The paper's structural conclusion doubles as a research agenda: designs converge where external constraints bind (tool categories, edit formats, execution isolation) and diverge exactly where no one knows the answer — context compaction, state management, multi-model routing. Deliberately, it reports no benchmark scores: architecture and capability are confounded, which is the point of §12.

§14Context management becomes a trainable skill

The taxonomy's biggest open question got its own line of attack. CAT — "Context as a Tool" arXiv:2512.22087 (Beihang/Manchester/Ubiquant; Findings of ACL 2026) reframes compaction from a passive heuristic the harness fires into an action the agent chooses: the context workspace is structured into stable task semantics, condensed long-term memory, and high-fidelity short-term interaction, and the agent calls a compression tool at milestones. The training story is the interesting part: CAT-Generator injects context-management actions into complete trajectories offline (averaging 87.4 steps and 4.22 management actions each; folded history compresses ~15.6K → ~4.7K tokens), producing 20K SFT instances that — together with 20K general instances — post-train Qwen2.5-Coder-32B into SWE-Compressor: 57.6% on Verified under a bounded context budget across up to 500 rounds, beating ReAct and static-compression baselines. For calibration, its own baseline table places SWE-agent-LM-32B at 40.2%. The result triangulates with §12's longitudinal finding (context-management changes are where harnesses break) and §15's production data (OpenHands' default condenser cuts API cost up to 2× with no performance loss). source-checked

§15Production SDKs and meta-agents

The OpenHands Software Agent SDK arXiv:2511.03690 (MLSys 2026, with ACM reproducibility badges) is the reference account of hardening a research scaffold: the V0 monolith redesigned into four decoupled packages (sdk, tools, workspace, agent-server) around event-sourced state — every event persisted (median 0.20 ms), deterministic replay (4.1 ms), crash recovery (7.4 ms) — with native sandboxing, lifecycle control, multi-LLM routing, and security analysis the paper's comparison table marks absent from the OpenAI, Claude, and Google agent SDKs. The receipts are operational as much as academic: over a 15-day production rollout, system-attributable failures dropped 61% (78.0 → 30.0 per 1,000 conversations; infrastructure errors 69.8 → 0.0 — V0's inter-pod HTTP auth failures eliminated by co-locating execution). On benchmarks it reports SoTA on 3 of 5 suites (GAIA 80.0%, Commit0 56.2%, SWE-bench Multimodal 44.1%) and 76.6% on Verified (Opus 4.5) against a 79.2% published SoTA — with the verification pass noting the +8.2-point V0→V1 gain holds for Sonnet 4.5 (64.6 → 72.8, attributed to extended-thinking support) but is exactly zero for Sonnet 4 (68.0 → 68.0): harness gains are model-conditional. corrected in verification

The Confucius Code Agent arXiv:2512.10398 (Meta + Harvard — our verification corrected an upstream mis-attribution to Meituan) pushes scaffold engineering to its logical end: a meta-agent that builds, tests, and refines coding agents in a build-test-improve cycle, instantiating CCA itself on an SDK organized around agent/user/developer experience with hierarchical working memory, persistent note-taking across sessions, and modular extensions. Headline: 59.0% Resolve@1 on SWE-Bench Pro with GPT-5.2 vs OpenAI's reported 56.0 under matched repos, backend, and tools; 74.6% on Verified with Claude 4 Sonnet. Its ablations are the cleanest public decomposition of scaffold value: advanced context management alone +6.6 points on a Pro subset; persistent notes lift repeat-domain resolve 53.0 → 54.4% while cutting turns (64 → 61) and token cost (104K → 93K). The paper's own version history is a period piece — v1's "state-of-the-art" claim was withdrawn within a day and re-posted with softened language, the GPT-5.2 headline arriving only in v6. corrected in verification

§16The minimalist counter-current

Against the SDK current runs a pointed argument from the people who started scaffold research. mini-SWE-agent github (SWE-agent team, Jun 2025) strips the harness to ~100 marketed lines of Python (190 on disk, our verification counted): bash as the only tool — no tool-calling API at all, actions parsed from fenced blocks — a strictly linear append-only history, and stateless subprocess.run execution (no persistent shell to babysit; trivially swappable for docker exec). It scored 65% on Verified with Claude Sonnet 4 at launch (July 2025) — five points shy of Anthropic's own unpublished scaffold — later 68%, and the README now claims >74% with Gemini 3 Pro. The stated rationale inverts 2024's premise: SWE-agent's special interfaces existed because chat-tuned models needed them; 2025 models are trained for agentic coding, so the scaffold can dissolve. Its second life is methodological — a model-centric baseline (the SWE-bench "bash-only" leaderboard) and a deliberately neutral harness for fine-tuning and RL research, "so you don't overfit to a specific agent scaffold." Terminal-Bench's Terminus 2 plays the same role from the eval side: a single-tool (headless tmux terminal) unopinionated loop used as the benchmark's neutral measuring instrument, defining task difficulty by frontier pass rates. The tension between §15 and §16 — hundreds of engineered components vs. a hundred lines — is the live design argument of 2026, and §12's data says both sides hold pieces of the truth: minimal scaffolds ride model capability; engineered ones still buy points at the margin, at real maintenance cost.

§17Synthesis, and what remains open

The three strands are one system now. Benchmarks became environments (Multi-SWE-bench ships Multi-SWE-RL; SWE-rebench feeds a leaderboard and Nebius's RL corpus; RepoLaunch builds both); models are trained inside specific harnesses on trajectories generated by other harnesses (SWE-smith in SWE-agent, SWE-Gym and SWE-Playground in OpenHands, daVinci-Dev on GLM-4.6-driven rollouts, Qwen3-Coder-Next distilled across six scaffolds); and harness skills are trained into weights (SWE-Compressor's compression tool, Qwen's anti-reward-hacking blocker). Evaluation, training, and harness are three views of the same executable-repository substrate.

• Contamination is an arms race, not a solved problem. The designated successor benchmark was itself reported ~30% broken within five months of its endorsement. Live/held-out/commercial splits and date-flagged leaderboards are the current defense; whether they hold as labs train on ever-fresher crawls is open.
• Rapid post-release score climbs are unexplained. Pro went ~23% → ~61.5%, Terminal-Bench 2.0 <65% → ~84.7%, within months. How much is capability versus scaffold tuning versus renewed leakage is exactly the question the Verified saga teaches us to ask early.
• Reward hacking scales with capability. Qwen's disclosure that agents learned to fetch gold patches over the network — requiring a dedicated blocker — suggests verifiable-reward integrity will need the same audit machinery UTBoost built for benchmarks.
• The mid-training stage is young. Two papers with strong numbers (58.5% without RL; +3–5 points portable across pipelines) but no shared definition of what belongs in the stage; interaction with RL is untested at frontier scale.
• Harness ablation science is thin. One longitudinal study (unreleased replication package), one taxonomy (no scores by design), vendor ablations (CCA's +6.6 context result) — controlled, cross-harness component attribution barely exists.
• Verifier gap. SWE-Gym's pass@16 (42.8%) vs verifier-selected best@16 (32.0%) gap, and DeepSWE's hybrid-verifier gains, mark reward modeling for agents as a first-order bottleneck for both TTS and RL.