Kimi K3: Open Frontier Intelligence

1. The first open 3T-class model. 2.78T total / 104.2B active MoE, native vision, 1M-token context, full weights released. The paper positions it as trailing only Claude Fable 5 and GPT-5.6 Sol overall, and ahead of every other model — open or closed — in its own suite. PAPER §1, §6
2. It's a bet that hybrid linear attention works at the frontier. 3 Kimi Delta Attention (linear, RNN-style state) layers per 1 Gated MLA (full attention) layer — 69 KDA + 24 MLA layers. This scales the Kimi Linear recipe ~58× in parameters, at exactly the moment MiniMax publicly retreated from hybrid attention with M2. NOTE
3. Three small architecture papers hide inside: Quantile Balancing (exact-quantile aux-free MoE routing, derived from LP duality), SiTU-GLU (soft-capped SwiGLU with a provable output bound), and Attention Residuals (layers attend over depth, not just sequence). PAPER §2
4. The vision encoder is trained from scratch with next-token prediction — no SigLIP/contrastive initialization — and matches the contrastively-initialized baseline while being more stable. A quiet challenge to five years of practice. PAPER §2.4
5. MXFP4 weights are not a deployment afterthought: quantization-aware training runs through SFT and RL, with rollout and training sharing the same quantized numerics — eliminating the train–inference mismatch that plagued 2025 RL stacks. PAPER §4.1.4
6. Post-training = 9 expert models distilled into one: {general, agentic, coding} × {low, high, max} reasoning effort, each trained with RL, then consolidated by Multi-Teacher On-Policy Distillation. Reasoning effort is a trained, budget-controlled behavior, not a decoding trick. PAPER §4.1
7. The RL environment story is the moat: 51.2M microVM sandboxes, a white-box harness that can impersonate Kimi Code / Claude Code / Codex / OpenClaw, knowledge-graph-driven task synthesis, kernel-optimization tasks with reward-hacking detectors, and mock Gmail/Notion/Slack ecosystems. PAPER §4.2, §5.3
8. Benchmarks: real wins, honest losses. Best-in-suite on BrowseComp (91.2%), SWE-Marathon (+7 pts over Fable 5), ProgramBench, MCPMark, τ³-Banking, OmniDocBench; first open model to top a major leaderboard (WebDev Arena). Clearly behind on HLE-Full, CritPt, GDPval Elo — research-level reasoning and knowledge-work polish. PAPER §6
9. Cost is the wedge: best BrowseComp score at $2.03/task — half of GPT-5.6 Sol, ~10× cheaper than Claude models at max effort, per their own cost curves. PAPER §6.4
10. What's missing: pre-training token count, GPU fleet size, training cost, data contamination discussion, and any alignment/safety-training section beyond an (unusually good) offensive-cyber evaluation with UK AISI + NIST CAISI. DeepSeek-V3 told us $5.576M; K2 told us 15.5T tokens; K3 tells us neither. NOTE

Moonshot's framing is explicit in the introduction: scaling has two axes — pre-training compute and test-time compute — and the open-source ecosystem has been living almost entirely on the second axis, with most open models parked at or slightly above the 1T-parameter regime while reasoning-RL methods improved. Their diagnosis: “open-source progress risks converging while the gap to the strongest proprietary systems widens.” K3 is the attempt to push both axes at once: a 3T-class pre-trained foundation and million-token agentic RL. PAPER §1

Four contribution buckets structure the report: (1) pre-training at the open frontier (architecture + data + ~2.5× scaling-efficiency gain over K2); (2) RL across domains and reasoning-effort levels, consolidated into one model; (3) infrastructure for 3T-scale training and 1M-token agentic RL; (4) the release itself — full weights on HuggingFace. PAPER §1

The organizing idea: scale information flow along sequence (hybrid KDA+MLA attention), depth (Attention Residuals), and width (Stable LatentMoE, 896 experts in a latent space). Each block = 3 KDA layers + 1 Gated MLA layer, each attention layer paired with a Stable LatentMoE FFN; one extra Gated MLA closes the backbone so the final layer is always global. PAPER §2, Fig. 2

2.1 · Kimi Delta Attention, and the one hyperparameter that buys back the Tensor Cores

KDA is a delta-rule linear attention with a channel-wise forget gate — the recurrent state S_t is updated as S_t = (I − β_t k_t k_tᵀ) Diag(α_t) S_{t−1} + β_t k_t v_tᵀ, i.e., decay each channel, erase along the current key direction, write the new key–value association. Queries/keys go through ShortConv → Swish → L2Norm; a low-rank projection plus per-head bias produces fine-grained decay logits per key channel. All inherited from Kimi Linear. PAPER §2.1.1

The K3-specific change looks tiny and is my favorite kind of systems-aware architecture edit. Kimi Linear mapped decay logits through an unbounded negative-Softplus, so the cumulative decay inside a chunk could underflow/overflow in finite precision; the chunkwise kernel therefore had to process diagonal 16-token tiles with explicit position-pair computations — the intra-chunk bottleneck. K3 swaps in a lower-bounded decay: g = g_min · Sigmoid(e^{A_h} z) with g_min = −5, so every retention factor stays above e^{−5} ≈ 6.7×10⁻³ and the reciprocal rescaling factor stays under e^{80} — inside BF16 dynamic range. Consequence: every tile, diagonal included, becomes a dense Tensor Core matmul. They also upgrade the output gate from low-rank to full-rank. PAPER §2.1.1, Fig. 3

The global layers are Gated MLA — DeepSeek-V2's latent-compressed attention (the KV cache stores a low-dimensional latent, up-projected on the fly), plus an input-dependent channel-wise output gate, plus NoPE: no positional encoding at all on the MLA layers. Position information lives entirely in KDA's recurrent decay. That's what makes 1M-token extension free of RoPE-rescaling surgery. They also keep attention output in FP32 during training to kill flash-attention's biased rounding error, redesigning the kernel to overlap the doubled output-tile footprint with KV staging. PAPER §2.1.2

2.2 · Attention Residuals — attention over depth

The pitch writes itself: standard residual streams compress all prior layers into one summed state — “a bottleneck reminiscent of RNNs over time” — so do to depth what attention did to sequence. Each layer l gets a learnable pseudo-query and attends (softmax-kernel, RMSNorm-stabilized) over the outputs of all preceding layers, mixing them with data-independent-per-layer but selective weights. Full form costs O(L²d) arithmetic (fine, L<100) but O(Ld) memory for keeping every layer output alive — painful under pipeline parallelism. So production K3 uses Block AttnRes: 12-layer blocks are summed into single block representations; attention runs over the ~8 block summaries; memory and communication drop from O(Ld) to O(Nd), with online-softmax merging of inter- and intra-block terms at inference. Empirically N≈8 recovers most of the benefit. PAPER §2.2

2.3 · Stable LatentMoE — 896 experts without the blowups

LatentMoE (cited to a Jan 2026 paper [32]) decouples model width from routed-expert width: shared experts keep the full 7,168-dim path; routed experts operate in a 3,584-dim latent space (0.5×). That halves per-expert communication and lets them afford 896 routed experts with 16 active — sparsity 56, roughly matching K2's 48 while doubling both pool and active count. At 2.8T scale two failure modes appear: exploding activations in the routed branch (the down-project → expert → up-project chain is “nearly four consecutive matrix multiplications” with no nonlinear guard), and load-balancing ~10³ experts beyond what existing aux-loss-free updates can handle. The three fixes: PAPER §2.3

• Normalized LatentMoE — an RMSNorm between expert aggregation and the up-projection. One norm, placed surgically; consistently improves validation loss.
• SiTU-GLU — SwiGLU with both branches soft-capped: [β₁ tanh(W_g x/β₁)] ⊙ Sigmoid(W_g x) ⊙ [β₂ tanh(W_u x/β₂)], β₁=4, β₂=25. First-order-identical to SwiGLU near the origin (the appendix expands the Taylor series), recovers SwiGLU as β→∞, and guarantees ‖output‖∞ ≤ β₁β₂ = 100. Unlike hard clamping, gradients never die at the cap. Overflow-proofing for a model that will spend its whole post-training life in 4-bit weights and 8-bit activations. PAPER §2.3.2, App. B
• Quantile Balancing (QB) — the star of the section, below.

Quantile Balancing, or: solving the routing LP instead of nudging it

DeepSeek-V3's aux-loss-free balancing adds a per-expert bias to router scores and nudges it each step with a fixed-size sign update — γ trades adaptation speed against oscillation, and at 896 experts the paper says it stops working well. QB replaces the nudge with the exact answer: choose each expert's bias so that, on the current batch's scores, exactly the target load q = mk/n tokens clear its threshold. That bias is a quantile — the (1−k/n)-quantile of each expert's margin distribution — and the appendix derives it as exact coordinate minimization of the LP dual of maximum-score balanced assignment (the BASE-Layers objective), with the V3 sign update revealed as SignSGD on the same dual. No learning rate, equilibrates in a few steps, and at deployment routing is just frozen-bias Top-k. Computing a global-batch quantile over millions of margins would be impractical, so they histogram per-expert margins (B≈1000 bins), all-reduce the counts (<1% of the comm cost of gathering raw margins), and read the quantile off the pooled histogram — error bounded by bin width, a few 10⁻³. PAPER §2.3.3, App. C–D

2.4 · MoonViT-V2 — vision without contrastive pre-training

K3 is natively multimodal — one backbone, one token stream, no post-hoc alignment stage. The encoder is a 401M, 27-layer ViT (RMSNorm, bias-free, factorized spatial/temporal attention for video, 2×2 pixel-shuffle, up to 3584×3584 px within the 1M context). The headline: it's trained entirely from scratch with next-token prediction. They show SigLIP-initialized encoders exhibit persistently higher, spikier gradient norms when jointly optimized with the LLM, while the from-scratch encoder stays stable — and final vision evals match. Their conclusion: “contrastive pre-training is unnecessary as an initialization for multimodal language models at scale.” PAPER §2.4, Fig. 6

2.5 · Per-Head Muon

K2 proved Muon at 1T scale (with MuonClip's QK-clip taming attention logits) PRIOR. K3 keeps Muon + weight clipping and refines it: Newton–Schulz orthogonalization is applied per attention head rather than to the full Q/K/V projection matrices, so large-momentum heads stop dominating the shared update direction. More balanced learning across heads, better stability at scale, and slightly cheaper (NS iterations on tall per-head blocks). PAPER §2.5

Data: four text domains (Web, Code, Math, Knowledge) plus a large vision corpus (captions, interleaved docs, OCR, perception, video, and — notably — “visual coding data”: code snippets paired with their rendered outputs across SVG, 3D, webpage, game, and CAD formats). The K2 rephrasing recipe returns: knowledge and math corpora are rewritten with perspective-diverse prompting and fidelity verification, a token-efficiency play. PAPER §3.1

What's conspicuously absent: the token count. K2 disclosed 15.5T tokens; DeepSeek-V3 disclosed 14.8T; Qwen3 disclosed 36T PRIOR. K3 discloses none, nor FLOPs, nor cluster size. The scaling-law section reports only a relative result: holding the K2 recipe as baseline, the K3 architecture+data changes deliver ~2.5× overall scaling efficiency (same validation loss at ~2.5× less compute, Fig. 7). PAPER §3.2 NOTE

Long context as curriculum, not surgery: pre-training starts at 8K, extends to 64K in a later phase, then 256K→1M during cooldown — a four-stage curriculum concentrating the expensive long-sequence compute into a small budget slice. Because MLA layers are NoPE, there's no RoPE-base retuning or YaRN step at any point. Long-context data gets a dedicated cleaning pipeline (exact+fuzzy dedup, perceptual video hashing, structural validation), natural long docs are upsampled, and — the key admission — “length alone does not confer long-range capability,” so they synthesize tasks whose answers require attending across the full 1M window. PAPER §3.3–3.4

The pipeline is a clean three-stage: SFT → RL-trained domain×effort experts → Multi-Teacher On-Policy Distillation (MOPD) back into a single model. PAPER §4.1

4.1 · SFT and the XTML template

SFT data: synthesized agentic trajectories from prior Kimi-series specialist models, multi-stage verification, human-in-the-loop annotation. All serialized in XTML — an XML-like template where angle brackets become reserved tokens [open]/[sep]/[close], making every structural boundary an explicit special token (no tokenization ambiguity, trivial constrained decoding). Assistant messages carry three channels — think, response, tools — explicitly inspired by OpenAI's Harmony format; thinking is preserved across turns in thinking mode; tool calls are indexed and typed with raw-text arguments so code isn't double-escaped. Options (reasoning effort, tool_choice, response_format) are injected as natural-language instruction messages rather than dedicated syntax — placed before the input messages if session-global (KV-cache friendly), after if per-request. PAPER §4.1.1, App. F

4.2 · RL — domains × effort levels

Rather than one RL run, they train nine experts: {general, agentic, coding} × {low, high, max} reasoning effort. Each domain spans a wide task spectrum (general: knowledge work, vision, search, faithfulness; agentic: long-horizon assistant, deep research, paragraph-level writing; coding: SWE, kernels, web dev). Training loops run “often over hundreds or thousands of tool calls and millions of accumulated context tokens.” Effort levels aren't prompt engineering: a per-problem token budget b₀(x) is estimated from the cold-start model, trajectories exceeding τ·b₀ get reward −1, and a stage-wise curriculum over the multiplier τ first trains a max-budget variant, then anneals τ down to carve out high- and low-effort experts. PAPER §4.1.2

Two mechanisms keep long-horizon RL trainable: partial rollouts (from K1.5/K2.5 — policy optimization proceeds when a fraction λ of trajectories completes; stragglers pause, persist via the sandbox layer, resume next iteration), and a per-token-regularized policy update that tolerates the resulting extreme off-policyness. For non-verifiable tasks, the Agentic Generative Reward Model keeps K2's tournament-style pairwise comparisons but forces the judge through a mandatory agentic protocol: read the artifact → generate a rubric → score each candidate against it → record scores in a scratchpad. Verbosity hacking is answered with a budget rule: exceed σ·ℓ₀ in length and you auto-lose the comparison. PAPER §4.1.2

4.3 · MOPD — distillation as consolidation

The nine experts are merged by on-policy distillation: the student samples; each token gets a dense reward clip(sg(log π_teacher/π_student)) from the domain-and-effort-matched teacher; the same RL infrastructure (partial rollouts included) trains against it. They tried finer-grained top-k distillation objectives and found no advantage. This is the Thinking Machines “on-policy distillation” recipe [75] operationalized at frontier scale, and the paper notes MiMo-V2 and DeepSeek-V4 as concurrent users of the pattern — multi-teacher OPD is apparently the 2026 consensus answer to “how do you ship one model that's good at everything without RL runs fighting each other.” PAPER §4.1.3 NOTE

4.4 · The quantization story nobody else has shipped

MXFP4 quantization-aware training runs through the entire post-training stage — SFT and RL — with MoE expert weights (the bulk of 2.78T) in MXFP4, activations in MXFP8, and attention/latent projections/shared experts/routers in higher precision. The kicker: during RL, rollout and training share the same quantization scheme, “eliminating the train–inference mismatch.” PAPER §4.1.4

4.5 · The draft model gets a loss function upgrade

The pre-trained MTP layer is fine-tuned into an EAGLE-3-style speculative draft (7-step unrolled, self-feeding after step one, fusing low/mid/high-level target features taken from the 1st, 4th, and final AttnRes blocks). Instead of the conventional KL surrogate they optimize the LK loss — the negative log of the token-level acceptance rate Σ min(p,q) itself, i.e., train directly for what speculative decoding actually pays for. Draft fine-tuning also runs under the deployment QAT config. PAPER §4.1.4

This section reads like the real moat. Six paradigms: PAPER §4.2

• Unified white-box harness. An agent harness decomposed into configurable modules — tool interfaces, system prompts, context management, skills, memories, subagents — that can be composed into “mainstream harnesses such as Kimi Code, Claude Code, Codex, OpenClaw, and Hermes, as well as entirely new ones.” During RL the harness configuration is randomized per task group, so the model can't overfit one tool schema or interaction protocol. This is anti-overfitting at the harness level, and it directly explains the eval section's cross-harness scores (K3 posts strong numbers under Claude Code and Codex, not just its own CLI).
• Knowledge-graph-guided task synthesis. A self-evolving concept DAG, expanded by web-searching agents from seed nodes until concepts are “sufficiently atomic”; task synthesis samples node combinations, retrieves real public materials, and generates tasks of chosen types against them. Coverage and granularity become controllable dials rather than emergent properties of a scraped dataset.
• Kernel-optimization tasks with anti-cheating. Single-op → fused mega-kernels across CUDA, Triton, CuTe DSL, Gluon, ThunderKittens, TileLang; rewards blend correctness (numerical threshold = zero reward if exceeded) and performance (0.5 at expert parity, →1 approaching the hardware roofline). A hacking-detection system explicitly penalizes CUDA-graph replay, input caching, and precision reduction — continuously extended as new exploits are observed. RL reward-hacking treated as a live adversarial security problem, documented in a tech report.
• Personal-assistant tasks in mock ecosystems. Faithful mock Gmail/Notion/Slack/Canvas implementations; multi-day persistent scenarios with interdependent event streams; up to thousands of tool calls and millions of context tokens per rollout; workspaces auto-constructed by web-searching agents.
• Autonomous Execution Tasks (AET). Goal + constraints + tool space + budget + independent verifier; no reference trajectories. Verifier types include black-box system replication (rebuild a hidden system by probing it — Fig. 10 shows K3 at 1.000 completion vs Opus 4.8 at 0.918 on a “Camera Repair Management System” replication), quantitative factor discovery, and tax auditing. Reward hacking mitigated by hidden verifiers held out from the agent and penalty-based rewards under limited submission budgets.
• Web-dev tasks with two-part rewards: deterministic checks (build/run/behavior; reward zeroed if the artifact fakes rather than implements) plus model judging that inspects source and interacts with the artifact. Rolled out under diverse scaffolds, again for cross-harness generalization.

The infra chapter spans training, RL, and serving. Highlights only; the paper rewards a full read here if you work on systems. PAPER §5

6.1 · Making KDA fast everywhere

• FlashKDA: CUTLASS-based chunkwise kernel overlapping intra-chunk compute with cross-chunk state propagation (token-parallel stages + head-parallel recurrence, independently scheduled); open-sourced, auto-dispatched inside flash-linear-attention.
• Intra-device context parallelism: for long prefill under tensor parallelism, segment state transitions are computed independently from S=0 across a rank's SMs and composed exactly afterward — an SM-level prefix scan.
• KDA Context Parallelism (KCP): the cross-device version. Vanilla linear attention composes across ranks by summing locally-generated states, but KDA's delta rule multiplies the incoming state by token-dependent matrices, so summation is wrong. KCP decomposes each rank's effect into (cumulative transition matrix, locally-generated state), both computable from local tokens only, exchanged with one fixed-size all-gather, then composed associatively — a prefix scan over matrix-affine maps. This is what makes 1M-token training tractable for the KDA layers. PAPER §5.1.2, Eq. 17

6.2 · 3T-class pre-training

• MoonEP: expert parallelism with perfect load balance via dynamically planned redundant experts. The proof in App. E shows a balanced plan always exists with at most E/R redundant experts per rank (essentially tight); reserving that many slots means planning always succeeds and training never stalls — contrast ECHO/UltraEP-style fixed caps that can hit infeasibility PAPER §5.2.1, App. E. Perfect balance buys three second-order wins: zero-copy communication (fixed S×K buffer vs S×K×R worst case), statically-known shapes (no per-layer host–device sync), and workload-aware expert-GEMM scheduling.
• Memory: a unified activation manager where recomputation/quantization/offload are pluggable per-tensor storage policies (most activations block-FP8 + offload); SonicMoE-style algebra to drop the routed-probs backward dependency; Block AttnRes computed once and shared; activation rebalancing across PP ranks via remote offload (Mooncake transfer engine); Pipeline ZeRO-2 gradient sharding with CPU-resident shards; P2P per-shard Muon orthogonalization instead of full-parameter all-gathers.
• Multimodal: large images split across CP ranks along the patch dimension (gather-KV attention); ViT forward/backward passes scheduled into pipeline bubbles, extending K2.5's Decoupled Encoder Process until the vision encoder is “largely free.”

6.3 · Million-token agentic RL

• External KV-cache pool: reusable long prefixes written back to CPU DRAM on eviction (write-back, not write-through), KDA states offloaded/prefetched in lockstep with their MLA blocks; training states spill to NVMe after each iteration to free DRAM for the pool.
• Auto-throttling rollout scheduler: admission concurrency driven by live KV-pressure signals rather than a fixed number tuned to average trajectory length.
• AgentENV: the sandbox layer (open-sourced, built with the kvcache-ai/Mooncake orbit). Firecracker microVMs (“agents should be able to mount disks, run containers, or even launch VMs” — container runtimes saw kernel panics from exploring agents); incremental checkpoint/resume at 133ms/49ms; pause (a paused sandbox costs nothing while the model thinks — “as much as 98% of the sandbox lifetime”), fork (side-effect-free reward judging), snapshot (error recovery); OverlayBD images with P2P transport, sub-second launch, 6.5× memory overcommit. 51,219,741 sandboxes / 1,505,678 images over K3's training and evaluation. PAPER §5.3.2

6.4 · Serving a hybrid at 1M context

The subtle problem: prefix caching for a hybrid model must jointly restore a paged, growing MLA KV cache and a fixed-size KDA recurrent state. Naive block-hash caching forces block granularity to 1024–6144 tokens (where KDA snapshots are affordable), making short-prefix reuse impossible. Their fix decouples the granularities: 512-token hash blocks inside physical pages for MLA matching, with KDA checkpoints persisted only at sparse hash endpoints (conversation-turn boundaries kept, intermediates recycled); a hit restores the longest boundary satisfying both, then resumes prefill from there. Plus: MTP×KDA speculative decoding via projected-input caching and state replay (independently proposed as ReplaySSM); a WarpDecode-based token-centric MoE decode kernel for memory-bound expert streaming; fleet-level cache-aware affinity scheduling (consistent hashing, primary + pre-assigned secondary cluster per session so failover re-prefill is spread across the fleet); and budget-based admission control so 1M-token bursts can't starve short requests (per-request cost spans three orders of magnitude). PAPER §5.4

7.1 · How to read their main table

Baselines: Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8, GPT-5.5 (xhigh), GLM-5.2 — all at max effort. Evaluation hygiene is unusually explicit: each model runs coding tasks under one of three harnesses (Kimi Code / Claude Code / Codex) — effectively each model on its home turf; third-party numbers are cited with as-of dates; footnotes count refusals and fallbacks per model (e.g., Fable 5 “hits fallbacks on 35% of [Terminal-Bench] GPU tasks,” refused 14 tasks on one internal bench; GPT-5.6 Sol results “include potential cyberguards”). PAPER §6.1.3

The pattern: K3 wins where long-horizon execution, tool orchestration, or breadth dominates (browsing, MCP tooling, marathon SWE, document/vision parsing, banking workflows), stays within a point or two on mainstream agentic coding, and loses cleanly where deep research-grade reasoning or knowledge-work judgment dominates: HLE-Full (−9.8 vs Fable 5 without tools), CritPt (23.4 vs Sol's 32.3), GDPval/AA-Briefcase Elo, OfficeQA, legal/finance agent suites. The authors say this out loud: “research-level reasoning remains a key direction for improvement.” PAPER §6.1.4

7.2 · Internal benchmarks, cyber, third parties, cost

The in-house suite (Table 3) is honestly labeled and mostly unverifiable from outside; the standout claim is the Kimi Webdev Bench blind expert study: K3 preferred over Claude Opus 4.8 at +31.0 points overall win−lose margin, with +59.1 on 3D/WebGL/shader tasks — both models run under the Claude Code harness, which is a nicely hostile choice of home field. PAPER §6.2, Table 4

The cyber evaluation is the closest thing to a safety section: Tier 1 (vulnerability discovery — “primarily associated with defensive security research”) found hundreds of candidates, ~70% of human-reviewed findings confirmed genuine, 16 previously unknown vulnerabilities across six projects, including two Linux-kernel findings (a remotely triggerable heap OOB write; a Dirty-COW-class RDMA permission-check regression). Tier 2 (end-to-end exploit development) solves 14/36 vs GLM-5.2's 8/36, with kernel-track success under one quarter. A joint UK AISI + NIST CAISI assessment is cited concurring: K3 beats GLM-5.2 on ExploitBench (32% vs 24%) but achieved arbitrary code execution on 0/41 tasks, trailing “frontier cyber-capable models.” Anthropic and OpenAI models refused the tasks and were excluded. Framing: “we regard our evaluation as a lower bound.” PAPER §6.2.2

Third-party scores (as of Jul 23, 2026, per the paper): Artificial Analysis Intelligence Index v4.1 = 57.1, #4 of 580 (behind Fable 5 at 59.9, GPT-5.6 Sol at 58.9); Vals Index #2 of 39 (74.7 vs Fable 5's 75.1); WebDev Arena #1 of 99 at 1,678 Elo — “the first open model to top this leaderboard” — Text Arena #8, Agent Arena #4. PAPER §6.3

Cost curves (Fig. 13): best BrowseComp score at $2.03/task (≈half of GPT-5.6 Sol, “an order of magnitude cheaper than the Claude models at their maximum effort”); Kimi Code Bench: −4.0 pts vs Fable 5 at 38% of its cost; GDPval: within 50 Elo of GPT-5.6 Sol at 13% lower cost, 2.6× cheaper than Fable 5. The strategic read: K3 isn't priced to beat the frontier on peak capability — it's priced to make the frontier's margins look silly on the 80% of tasks where 95% of the quality suffices. PAPER §6.4 NOTE

• GPU kernel optimization (24h budget/task, sandboxed): cut AttnRes latency 283.6→114.4 ms, DSA and KDA runtimes by 55–74%, “matched Claude Fable 5 (with fallback) and substantially outperformed” the others; +59.7% vs the FLA Triton baseline on the trajectory plot. The aside that an early K3 checkpoint “was already handling most of our kernel optimization work during late-stage development” is the recursive-self-improvement sentence of the year, stated with zero drama. PAPER §7, Fig. 14
• MiniTriton: a compact Triton-like compiler (Python frontend, MLIR-like IR, PTX codegen) plus a PyTorch-like tensor library with autograd, NCCL distributed primitives, and a matched-within-10⁻⁴ GPT training run — beating torch eager and torch.compile in geometric mean on its benchmark suite on an L20, its matmul reaching ~90% of measured machine roof vs cuBLAS at large shapes. Open-sourced. PAPER §7, Fig. 15
• Chip design: a 48-hour autonomous run producing RTL for “nano-KPU,” an INT4 inference chip for a nano-scale model of K3's own architecture — timing closed at 100 MHz on Nangate45 with open-source EDA, 1.46M cells, 8,700 tok/s RTL-simulated. Explicitly “an early proof of concept.” PAPER §7
• Research & knowledge work: reproducing I-Love-Q universal relations across 20+ astrophysics papers (~2h vs an estimated 1–2 researcher-weeks); a 42-year AI-ASIC industry site from 87 quarterly reports + 99 PDFs over 120 refinement rounds; GWTC-5 analysis of 391 gravitational-wave events with 20+ concurrent subagents. A 3Blue1Brown-style motion-graphics video explaining its own architecture, edited from 56 source clips. PAPER §7

The comparison table below mixes eras deliberately: it's how the open-frontier recipe drifted, 2024→2026. Specs for non-K3 models come from their own reports PRIOR; K3's from this paper PAPER. I've deliberately avoided quoting old benchmark scores — they go stale in months and cross-report numbers are never apples-to-apples; architecture and disclosure are the durable comparisons.

Non-K3 rows from each model's own report/blog, my training knowledge — treat exact figures as “as reported by the vendor, circa release.” Llama 4 and Qwen3-Next entries are the least standardized (blog-sourced; some figures approximate). PRIOR

9.1 · The hybrid-attention bet — K3 doubles down where MiniMax folded

This is the sharpest cross-report storyline of 2025–26. MiniMax-01 was the first to ship linear-hybrid attention at a 456B flagship (7 linear : 1 softmax) and claimed 4M context PRIOR. Ten months later, MiniMax's M2 reverted to full softmax attention, with the team publishing an unusually candid explanation: linear hybrids showed real quality gaps on agentic and long-chain reasoning workloads, and the surrounding infrastructure (kernels, caching, serving) wasn't mature enough to bank the theoretical efficiency PRIOR. Meanwhile Qwen3-Next quietly validated a 3:1 Gated-DeltaNet hybrid at 80B PRIOR, and DeepSeek took a third road entirely — trained sparse attention (NSA, then DSA in V3.2-Exp) that keeps softmax semantics but prunes the compute PRIOR.

K3 is a direct empirical answer to the M2 objections, point by point: quality gaps → agentic/long-horizon RL is exactly where K3 posts its best-in-table numbers (BrowseComp at full 1M, SWE-Marathon); infra immaturity → FlashKDA, KCP, the KDA-aware prefix cache, and the decode-time state-replay design are published and partly open-sourced; prefix-caching incompatibility → §5.4.1's decoupled hash-block/checkpoint scheme reaches “the same generality as full-attention models.” What K3 does not publish is the counterfactual: no ablation of K3-with-full-MLA at matched compute, so the strongest available evidence that the hybrid itself is a win remains Kimi Linear's 48B-scale ablations PRIOR. The honest summary: K3 proves hybrid linear attention is viable at the open frontier; whether it's optimal is still open. NOTE

9.2 · MoE design — the sparsity plateau and the latent turn

V3 → K2 → K3 expert counts go 256 → 384 → 896, active experts 8 → 8 → 16, sparsity 32 → 48 → 56. The marginal move in K3 isn't raw sparsity (K2's report already argued sparsity scaling laws motivated 48) — it's the latent routed space: halve the routed width, double the expert count and actives for similar cost, then spend engineering (RMSNorm placement, SiTU-GLU, QB) making it trainable. Compare Qwen3-Next's 512-expert ultra-sparse design at the small end PRIOR — the field is converging on “more, smaller, cheaper experts” from both ends of the scale axis. Meanwhile gpt-oss sits at the opposite pole (128 big experts, top-4) — optimized for single-GPU deployment rather than fleet economics PRIOR. NOTE

9.3 · Optimizers — Muon's scale ladder is now complete

Moonlight (16B) → K2 (1.04T, MuonClip, zero loss spikes over 15.5T tokens) → K3 (2.78T, Per-Head Muon + weight clipping) PRIOR PAPER §2.5. No other frontier lab has published a non-Adam optimizer at anything like this scale; every other report in the table above says AdamW or nothing. If you want a single unambiguous differentiator of the Kimi series' reports, it's that they keep publishing optimizer research the others treat as proprietary. NOTE

9.4 · Post-training paradigms — from “one RL run” to “an economy of experts”

Two convergences worth naming. First, effort as a product surface: gpt-oss exposed effort as a prompt-level switch; K3 turns it into a trained capability with its own RL curriculum and per-domain budget tuning — and the eval section then exploits it (score-vs-cost curves at low/high/max). Second, on-policy distillation as the consolidation layer: the Thinking Machines recipe went from blog post (Oct 2025) to the backbone of at least three frontier-scale pipelines (K3, MiMo-V2, DeepSeek-V4, per this paper's citations) in under a year — remarkably fast idea-to-frontier latency for a training-methods idea. NOTE PAPER §4.1.3

9.5 · Report culture — what gets disclosed, 2024→2026

Reading K3 against its peers as documents, three trends:

• Systems sections are eating the reports. V3 spent ~12 of 50 pages on infrastructure and was considered infra-heavy; K3 spends roughly a third of its body there, with proofs in the appendix and five open-sourced repos (FlashKDA, MoonEP, AgentENV, MiniTriton, nano-kpu). The open-weights competition is increasingly fought on serving economics, and the reports show it. NOTE
• Third-party evaluation is now built-in. K3 cites Artificial Analysis, Vals, arenas, and two governments' AI-safety institutes inside the launch report, with as-of dates — a norm K2 started (AA citations) and K3 completes. Compare 2024, when vendor tables were the whole story. The flip side: Elo-style numbers drift the day after publication, and the paper acknowledges it in a table footnote. PAPER §6.3
• Compute disclosure has gone backwards. V3's 2.788M H800-hours / $5.576M line item was a flex that reset expectations; K2 still gave tokens (15.5T); K3 gives neither tokens nor GPUs nor dollars — only the relative 2.5× efficiency claim. Whether the cause is export-control optics, competitive secrecy, or both, the most-open-ever model ships with the least-open-ever training bill. NOTE

1. No tokens, no FLOPs, no cost. The 2.5× scaling-efficiency claim is measured on held-out validation loss under their own data mix, and the absolute compute behind K3 is undisclosed. “2.5× more efficient than K2” and “how much did K3 cost” are independent questions; only the first is answered.
2. Key ablations live elsewhere or nowhere. AttnRes ablations → unpublished preprint [57]. KDA-vs-full-attention at scale → Kimi Linear (48B). LatentMoE optimality → a Jan 2026 paper [32]. The report is a systems integration proof, not an ablation study; almost no component is ablated at K3 scale in this document.
3. Benchmark provenance is mixed. Several headline wins are on benchmarks Moonshot created or co-created (Kimi Code Bench 2.0 is internal; PerceptionBench is theirs; SWE-Marathon and FrontierSWE are new 2026 suites). The report is unusually careful about labeling this, but “best on our own benchmark” rows deserve the discount you'd apply to any vendor.
4. Contamination goes unmentioned. No decontamination methodology for the public benchmarks — notable for a model whose task-synthesis pipeline crawls the public web at scale, including the domains the benchmarks live in.
5. Harness-fallback asymmetries. Competitor scores carry footnotes about refusals, fallbacks, and “potential cyberguards” — honest, but it also means several K3-vs-frontier gaps partially measure willingness, not capability. Terminal-Bench GPU tasks where Fable 5 falls back 35% of the time are a capability win for K3 only under the assumption the fallbacks weren't policy-correct.
6. Safety is one (offensive) chapter. There is no alignment-training, refusal-policy, or misuse-mitigation section for an open-weights release at claimed near-frontier capability. The cyber eval is excellent and externally corroborated, but it evaluates the hazard, not the mitigation.
7. 1M context ≠ 1M comprehension. AA-LCR (74.7, best-in-table) and BrowseComp-at-1M are encouraging, but the long-context evidence is thinner than the long-context engineering. No needle-grid, no RULER-style sweep, no per-depth analysis appears in the report.

• Does the hybrid hold at decode-heavy workloads in the wild? Community serving reports (vLLM/SGLang issues, latency benchmarks at 100K+ context) over the next quarter will tell us more than Table 2 does. Watch whether the KDA-aware prefix cache design gets reimplemented outside Moonshot.
• Will the Attention Residuals preprint substantiate the depth-attention claim? And will anyone reproduce a benefit at ≤10B scale, where most of the community lives?
• Is from-scratch NTP vision training reproducible off-frontier? If yes, SigLIP-style towers quietly retire. If no, it's a scale-gated result worth knowing the threshold for.
• How does K3 compare to DeepSeek-V4 head-to-head? Both reports now cite each other's techniques (MOPD; million-token context) but neither benchmarks the other. The first neutral third-party bake-off (AA, Vals, arenas over the next month) is the real open-frontier scoreboard.
• Does Quantile Balancing become the new default? It's simple, hyperparameter-free, and the derivation is public — I'd expect it in the next wave of open MoE reports; if it doesn't show up, that's evidence the sign-update rule is better than this paper suggests at ordinary expert counts.
• What does “Kimi K2.5 Agent Swarm” + K3's Swarm Bench lead imply about the next report? The intro frames parallel agent coordination as the third scaling axis; K3's best internal margins are on swarm-style orchestration. K4 (or K3.5) as a multi-agent-native release seems like the obvious trajectory.

Ordered by marginal value if you've just read K3. Links are to the canonical versions; scores quoted anywhere above should be re-verified before reuse — this field's numbers rot in months.

1. Kimi K2: Open Agentic Intelligence (Jul 2025)The direct predecessor; MuonClip, agentic data synthesis, self-critique rubric rewards. Table 1 of K3 is a diff against this.
2. Kimi Linear: An Expressive, Efficient Attention Architecture (Oct 2025)Where KDA and the 3:1 hybrid were actually validated with ablations. K3 §2.1 is this paper, scaled 58×.
3. DeepSeek-V3 Technical Report (Dec 2024)Still the reference open-frontier report: MLA, aux-loss-free routing (the thing QB refines), FP8 training, the $5.576M line K3 declines to match.
4. LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts (Jan 2026)The latent-expert idea K3's §2.3 stabilizes. Read before judging the 896-expert design.
5. Qwen3 Technical Report (May 2025)The thinking-budget/mode-fusion alternative to K3's trained effort levels; 36T-token disclosure as a contrast in openness.
6. MiniMax-01: Scaling Foundation Models with Lightning Attention (Jan 2025)The first linear-hybrid flagship — then read MiniMax's M2 release notes (Oct 2025) for the retreat K3 is answering.
7. Thinking Machines — On-Policy Distillation (Oct 2025)The consolidation recipe behind MOPD, in its original, readable form.
8. gpt-oss model card (Aug 2025)MXFP4 open weights + Harmony channels + effort levels — three K3 design choices have visible ancestors here.
9. Kimi k1.5: Scaling Reinforcement Learning with LLMs (Jan 2025)Partial rollouts and length-budget RL originate here.
10. Jianlin Su — Travels in MoE 6: Load Balance via Optimal Assignment (Feb 2026, Chinese)The blog post Quantile Balancing is credited to. Worth the machine translation.
11. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (Jun 2026)The other 2026 million-token open flagship; the natural head-to-head nobody has run yet. (Cited by K3; I haven't read it — it postdates my training.)
12. GLM-5: from Vibe Coding to Agentic Engineering (Feb 2026)Lineage of GLM-5.2, the only open baseline in K3's tables. (Same caveat — cited, not read.)

How these notes were made, so future-me knows how much to trust them:

• Primary source: the full 47-page PDF of arXiv 2607.24653v1 (downloaded 2026-07-28), read end to end including appendices A–F and the reference list. All PAPER claims trace to specific sections; benchmark numbers were transcribed from Tables 1–5 and Figures 1, 13, 14.
• Comparisons: from my training knowledge of the cited pre-2026 reports (DeepSeek-V3/R1, Qwen3, Qwen3-Next, MiniMax-01/M1/M2, Llama 3/4, gpt-oss, GLM-4.5, Kimi K1.5/K2/Linear, Moonlight, EAGLE-3, Harmony). These are marked PRIOR and were not re-verified against live sources for these notes; architecture facts are stable, but any quoted figure should be re-checked before being repeated.
• Blind spots: models released after Jan 2026 (Claude Fable 5, GPT-5.6 Sol, Opus 4.8, GPT-5.5, GLM-5.2, DeepSeek-V4, K2.5/K2.6, MiMo-V2, Inkling, Step 3.5 Flash) are known to me only through this paper's citations of them — I cannot independently characterize them. Third-party scores (AA, Vals, arenas) are as-of Jul 23, 2026 per the paper and have certainly drifted since.
• Known reliability gradient: strongest on what the paper says (verbatim source in hand); strong on pre-2026 architecture lineage; weakest on cross-model benchmark equivalence and anything Elo-shaped.

Companion documents in this folder: LLM coding-agents survey (2025–26) · Recursive self-improvement deep study