Last week, a Baseten inference engineer who goes by @waterloo_intern published a technical blog post titled "22,580: From GPT-2 to Kimi K3, Explained." It hit 2.4 million views in days.

He didn't write a press release. He wrote runnable PyTorch code — starting from GPT-2's attention block, stepping through every architectural change, explaining one problem and one cost per iteration. It's the best transformer lineage explanation I've seen.

I devoured his post, then cross-checked the key claims against 5 original papers. Here's the full picture.

The 22,580x Number

In February 2019, OpenAI released GPT-2 — 124M parameters. Seven years later, Moonshot AI open-sourced Kimi K3 — 2.8T parameters. You could fit 22,580 GPT-2s inside one Kimi K3.

But this isn't a "throw more compute at it" story. It's a story about how we store, update, and retrieve memory.

Starting Point: GPT-2 

class Block(nn.Module):
    def forward(self, x):
        x = x + self.attn(self.ln_1(x))
        x = x + self.mlp(self.ln_2(x))
        return x

Enter fullscreen mode Exit fullscreen mode

Every time the model generates a new token, it recomputes Q, K, V projections for all historical tokens, then runs an O(N²) softmax attention. K and V from tokens 1 through N-1? Thrown away. Token N+1 arrives? Recompute everything.

That's why KV Cache was invented.

KV Cache: Store It, Don't Recompute

Simple idea: cache the already-computed keys and values. For the next token, new Q only needs one dot product against the cached K.

Problem solved — but a new one created. KV cache grows linearly with sequence length. At 1M tokens × d_model × layers, that's dozens of GB of VRAM. Every decoding step reads all of it from HBM.

The bottleneck isn't compute. It's memory bandwidth. This is the key to understanding every improvement that follows.

Linear Attention: Fixed-Size Memory

Can we compress O(N²D) into O(ND²)?

The idea: replace softmax with a feature map.

# Standard softmax (must materialize N×N first)
attention = softmax(QKᵀ / √d) × V

# Linear attention (fold K and V first)
Q' = ELU(Q) + 1
K' = ELU(K) + 1
output = (Q'(K')ᵀ × V') / (Q'(K')ᵀ × ones)

Enter fullscreen mode Exit fullscreen mode

Now we can compute K'^T × V' first — a fixed-size D×D matrix — then multiply by Q'. Historical KV information gets "folded" into a constant-size state matrix. Cache no longer grows with N.

The cost? ELU+1 is an approximation of the softmax kernel. Expressiveness drops. But on long-context tasks, this tradeoff is often worth it.

Side Note: FlashAttention Didn't Do This

A common confusion point. Ali's post mentions "in 2020, there was no FlashAttention" — but FlashAttention and linear attention solve fundamentally different problems.

FlashAttention (Tri Dao, NeurIPS 2022) didn't change the attention algorithm. It optimized the GPU IO pattern — tiling the N×N matrix so it never fully lands in HBM. It makes softmax faster, but it's still O(N²).

Linear attention redefined the algorithm itself — replacing softmax with a feature map, going from O(N²D) to O(ND²).

One optimizes IO. One changes the algorithm. Orthogonal.

DeltaNet: You Can Now Edit Memory

Linear attention has a fatal flaw: it can only add, never update.

Every new token piles onto the state: S = S + K'^T × V'. Information only grows. It's like a notebook where you can only write — never erase or correct.

DeltaNet (Songlin Yang et al., NeurIPS 2024) fixes this.

The core idea traces back to Schmidhuber's "Fast Weight Programmers" from the 1990s, later formalized by Schlag et al. (ICML 2021) linking linear attention to fast weights. DeltaNet's approach: before writing, read what this key position currently stores.

v_old = k @ S_old           # read current value at key
delta = v_new - v_old       # compute the delta
S_new = S_old + k^T @ delta # write only the difference

Enter fullscreen mode Exit fullscreen mode

If v_new == v_old, delta is zero — nothing changes. If completely different, delta equals v_new — equivalent to an overwrite. Everything in between is a smooth interpolation.

This is the delta rule: precise memory updates instead of blunt accumulation.

The Parallelization Trick (Ali Said This Took Him 7 Hours to Understand)

DeltaNet's state update is strictly sequential — each step depends on the previous S. It looks impossible to parallelize. But it is.

The method: split the sequence into chunks of size C. Within each chunk, use normal masked attention (GPU-parallel). Between chunks, use the state matrix + one matmul (Q @ S). A Householder transformation reparameterizes the delta updates so all deltas within a chunk can be computed at once.

Complexity: fixed cost 2LD² (state maintenance) + variable cost 2LCD (intra-chunk attention). Bigger C = more variable cost but better GPU efficiency. In practice, C=64 or 128 works best — FLOPs aren't the only metric; tensor core utilization matters too.

Gated DeltaNet: Now You Can Forget

DeltaNet can precisely edit individual key-value pairs. But what about forgetting at scale?

Imagine reading through all documents about Topic A in a 1M-token context, then switching to Topic B. The model should ideally "forget" Topic A to free up capacity for B.

DeltaNet can overwrite specific entries but can't bulk-decay global memory. Mamba-2 (Dao & Gu, ICML 2024) can:

cache = α × S_old + S_new

Enter fullscreen mode Exit fullscreen mode

α is a gating value between 0 and 1, uniformly decaying all old memories. This is the core of Mamba-2's "State Space Duality" theory — softmax attention and SSMs are mathematically the same thing expressed differently. Mamba-2 unified them through gating.

Gated DeltaNet (Songlin Yang et al., ICLR 2025, NVIDIA) merges DeltaNet's delta updates with Mamba-2's gating:

S_new = α × S_old + k^T @ delta   # decay first, then precise write

Enter fullscreen mode Exit fullscreen mode

α=1 is pure DeltaNet. α=0 is a memory wipe. Everything in between both forgets and updates.

The crucial mechanism: a token written at timestep x, read at x+t, has been through t rounds of cumulative α decay (αₓ × αₓ₊₁ × … × αₓ₊ₜ). Information written at different times forgets at different speeds — recent items decay little, distant ones may be fully gone.

KDA: Kimi's Secret Sauce

Kimi Linear (Moonshot AI, arXiv 2510.26692, October 2025) — the predecessor to K3 — refined the idea further: from scalar gating to per-dimension gating.

Gated DeltaNet uses α as a single scalar. One number controlling forgetting speed for all memory dimensions. KDA turns α into a vector (or matrix). Each dimension independently controls its own forgetting rate. Which concepts to retain, which dimensions to decay — the model learns it.

Key data from the paper:

Metric Full MLA Kimi Linear
KV Cache 100% -75%
1M-context decode throughput Baseline 6x
Short-context performance Baseline Outperforms
RL scaling Baseline Outperforms

This is the first time linear attention outperforms full attention under fair comparisons across all scenarios. Not just long context — short context too. And that "-75% KV cache" translates directly to inference cost savings.

I verified this claim directly against the paper's abstract: "for the first time, outperforms full attention under fair comparisons across various scenarios." Not marketing. Core conclusion.

K3's Final Hybrid Architecture

K3's technical report (arXiv 2607.24653, July 2026) confirms:

  • 3/4 layers use KDA (linear), 1/4 use gated MLA (full softmax)
  • MoE routing: 896 experts, 16 active per token (~1.8%), Quantile Balancing for load distribution
  • Attention Residuals: layers can "look back" at specific earlier-layer representations
  • MXFP4 weights + MXFP8 activations, quantization-aware training

K3 isn't a pure linear-attention model. It's a hybrid system: KDA handles the bulk processing (cheap, fast, fixed-state), while periodic softmax layers do precision retrieval — recovering details that linear compression might lose. Exactly the tradeoff Ali analyzed: linear attention + periodic softmax retrieval.

So What Does 22,580x Actually Mean?

Ali's conclusion is sharper than anything I could write:

From GPT-2 to Kimi K3, each generation's core improvement wasn't "more parameters" — it was redesigning how memory is accessed: how you store, how you forget, how you retrieve.

The evolution in one table:

Stage Representative Memory Mechanism Problem Solved
1 GPT-2 + KV Cache Full cache, O(N) growth Eliminate recomputation
2 Linear Attention Fixed state, O(D²) Cache doesn't grow with N
3 DeltaNet Delta updates, editable Can't update → can update
4 Gated DeltaNet Gating + delta, forgettable Can't forget → can forget
5 KDA / Kimi K3 Per-dim gating + hybrid Linear surpasses full attention

Parameters grew 22,580×. But if that's all you see, you missed the entire story.

What This Means for You

If you're doing long-context work (code review, document analysis, multi-turn agents), attention architecture directly impacts your cost and output quality.

  1. Cost isn't fixed. Same 1M-token context: KDA's KV cache is only 25% of full attention's. Lower memory bandwidth pressure, significantly better latency and throughput.

  2. Long ≠ expensive. K3's 1M-token context window isn't "brute-forced." 75% of work goes through the linear path.

  3. Hybrid is the trend. Linear won't replace softmax — they'll complement each other. Precision retrieval via softmax, bulk processing via linear. This paradigm will spread.

Key Takeaways

  1. 22,580× parameter growth is surface-level. The real story is memory management: full cache → fixed state → precise writes → adaptive forgetting.
  2. The lineage is clear: DeltaNet (NeurIPS 2024) → GatedDeltaNet (ICLR 2025) → Kimi Linear (Oct 2025) → Kimi K3 (Jul 2026).
  3. KDA is the first to surpass full attention in fair comparisons — across short, long, and RL scaling scenarios. Not a "cheaper alternative."
  4. K3's 75% linear / 25% softmax hybrid is an engineering optimum, not a paper-only construct.
  5. Next time you evaluate a model: ask about attention architecture, not parameter count. It matters more.

References